Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

Evaluating and Improving Large Language Models for Competitive Program Generation

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On 80 problems from nine 2024 ICPC/CCPC regional contests, a state-of-the-art reasoning LLM given a basic prompt fully solves only 5; a taxonomy-guided repair-and-regeneration framework raises the count to 46.

desk verdict A useful leakage-controlled benchmark and a plausible 5/80 baseline, but the 46/80 improvement claim is confounded by human-written solution hints in Phase 3 prompts. read the letter →

arxiv 2506.22954 v1 pith:VZWNL7GY submitted 2025-06-28 cs.SI cs.SE

Minnan Wei , Ziming Li , Xiang Chen , Menglin Zheng , Ziyan Qu , Cheng Yu , Siyu Chen , Xiaolin Ju This is my paper
classification cs.SIcs.SE
keywords competitiveprogramminglargelanguagemodelscodegenerationerrortaxonomypromptengineeringprogramrepairDeepSeek-R1empiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how well a strong reasoning LLM can write correct programs for genuinely new competitive programming problems, and whether its failures can be systematically fixed rather than worked around. The authors collect 117 problems from nine regional ICPC/CCPC contests held in 2024, filter them down to 80, and test DeepSeek-R1 with a deliberately basic prompt; only 5 of the 80 generated solutions are fully accepted by online judges. They then classify the 75 failures with a two-branch hierarchical error taxonomy and design a three-phase improvement framework — error diagnosis, multi-turn dialogue repair, and information-augmented regeneration — that raises the number of accepted problems to 46 of 80. If the paper is right, it shows that today's strongest open reasoning models remain far below contestant level on fresh contests, and that most of their mistakes are diagnosable and repairable with structured guidance.

What carries the argument

The load-bearing mechanism is the hierarchical error taxonomy, defined as a two-branch classification — general errors (design, boundary, condition, data type, syntax, input/output) and algorithm-specific errors (mathematical, greedy, graph, recursion and divide-and-conquer, dynamic programming, search) — built by two annotators on 60 of the 75 failed programs with Cohen's kappa agreement of 0.86. The taxonomy does the causal work in the improvement pipeline: Phase 1 locates and classifies the fault, Phase 2 maps each error class to one of seven repair strategies (from full algorithm regeneration down to I/O format fixes) in multi-turn dialogue, and Phase 3 regenerates code from scratch with a prompt template that injects human-authored scaffolding. The taxonomy converts an opaque Wrong Answer verdict into an actionable repair target, which is what lets the pipeline fix 41 of 75 failures.

What would settle it

Run the identical basic-prompt evaluation and the identical three-phase framework on the 37 problems removed by the four filters, or on a fresh unfiltered sample of 2024–2025 ICPC/CCPC problems. If acceptance rates on those problems stay near zero both before and after the pipeline, the headline results are specific to the curated subset; if Phase 3 is rerun with the human-written algorithm hints removed from the prompt template and most of the gain disappears, the improvement is driven by the injected hints rather than by the repair dialogue on its own.

Watch

Extended reading notes

Core claim

The paper's central claim is empirical: on a leakage-resistant benchmark built from 80 problems in nine 2024 ICPC/CCPC regional contests, DeepSeek-R1 given a task-specific basic prompt fully solves only 5 problems, with 63 Wrong Answer, 5 Time Limit Exceeded, and 7 Compile Error verdicts, and correctness collapses by difficulty (4 of 13 warm-up, 0 of 36 bronze, 1 of 31 silver) and by complexity (4 of 49 single-algorithm, 1 of 31 multi-algorithm). The paper further claims that these failures are systematic rather than random: 63.3% fall into general error categories, led by design errors (28.6%) and boundary errors (15.5%), while 36.7% are algorithm-specific, led by mathematical, greedy, and graph errors. Its final claim is that a taxonomy-driven improvement framework — Phase 1 diagnoses and classifies the fault, Phase 2 applies strategy-specific multi-turn repair prompts, Phase 3 regenerates from scratch with an information-augmented prompt carrying problem modeling, algorithm ideas, implementation details, and constraints — converts 41 of the 75 initial failures into accepted solutions, raising the total from 5 to 46 of 80. The authors present this as showing that current reasoning LLMs are far from contestant level on fresh problems, and that most of their failures are identifiable and repairable with structured guidance.

Load-bearing premise

The load-bearing premise is that the 80 problems left after filtering — which deliberately exclude gold-level problems, image-based geometry, long descriptions, and non-submittable items — fairly stand in for 'real-world competitive programming,' so that the 5/80 baseline and the 46/80 improvement describe contests rather than a specially selected, LLM-aligned subset.

Editorial extensions

If this is right

  • Fresh contest problems give a leakage-resistant test: because these 2024 contests postdate the model's training data, the 5/80 baseline cannot be explained by memorization, and the benchmark is reusable for evaluating later models.
  • Most failures are repairable: 41 of the 75 initially failed problems reach acceptance through the pipeline, so the basic-prompt score understates the model's achievable performance rather than marking its ceiling.
  • Difficulty remains the bottleneck: after the full framework only 8 of 31 silver-level problems pass, so the gains concentrate in warm-up and bronze problems.
  • Multi-algorithm problems stay the hardest class (from 1/31 to 13/31 accepted), indicating that combining several algorithmic techniques is a distinct failure mode that prompts and repair do not fully resolve.
  • Design-related errors dominate the taxonomy (28.6%), pointing improvement efforts toward algorithm selection and requirement comprehension rather than syntax or input/output details.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 5/80 and 46/80 figures describe a curated slice of contests: the four filters deliberately removed gold-level, image-based geometry, and long-description problems, and a random unfiltered sample of 2024 ICPC/CCPC problems would almost surely yield both a lower baseline and a lower final acceptance count.
  • The improvement is human-assisted, not autonomous: Phase 3 injects author-written algorithm hints into the regeneration prompt, so the 41-problem gain bundles the taxonomy's guidance with external algorithm knowledge; a control run with the hints stripped out would separate the two contributions.
  • The taxonomy's dominant error classes — incorrect algorithm, off-by-one and boundary handling, greedy-choice validity, and DP state definition — read like a checklist of automated critic signals that could be fed back into training or inference-time search, making the diagnostic structure testable beyond this one model and contest season.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper reports an empirical study of LLM-based competitive program generation. The authors collect 117 problems from nine 2024 ICPC/CCPC regional contests, apply four filtering criteria (excluding gold-level problems, geometry problems with visual images, problems with difficult-to-process descriptions, and non-submittable problems), and retain 80 problems. Using DeepSeek-R1 at temperature 0.7 with a three-part basic prompt, they submit one generated solution per problem to online judges and report 5 AC, 63 WA, 5 TLE, and 7 CE. They construct a hierarchical error taxonomy from 60 non-AC programs (development set) and validate it on 15 programs, reporting Cohen's kappa of 0.86. They then propose a three-phase improvement framework: Phase 1 diagnoses and classifies errors, Phase 2 applies multi-turn dialogue-based repair, and Phase 3 regenerates solutions from an information-augmented prompt template. The framework reportedly increases the number of AC solutions from 5 to 46 out of 80 problems. The paper claims contributions of a new benchmark, an initial evaluation, an error taxonomy, and an improvement framework.

Significance. If the basic-prompt evaluation is reliable, the 5/80 result is a useful and relatively timely data point on a state-of-the-art reasoning LLM for 2024 ICPC/CCPC problems, and the shared benchmark, scripts, and OJ-based evaluation are concrete reproducibility assets. The error taxonomy is carefully constructed with two annotators, a third adjudicator, and a reported inter-annotator agreement of 0.86. However, the significance of the improvement result is substantially limited by a confound: the Phase 3 regeneration prompt template embeds human-authored algorithm ideas and implementation details, so the measured gain from 5 to 46 AC is not attributable to the LLM's own improvement. The single-sample stochastic evaluation and the deliberate benchmark filtering also bound the strength of the headline numeric claims. The paper's main value is as a reproducible evaluation and error-analysis artifact rather than as a validated method for improving LLM competitive-program generation.

major comments (5)
  1. [§5.3, Figure 18] The Phase 3 regeneration prompt template includes fields [Problem Modeling], [Algorithm goal], [Algorithm ideas], [Implementation details], and [Constraints], and the provided example for CCPC-SF Problem F specifies a complete solution approach, including 'Use a linear sieve to get minimum prime factors', 'maintain two DP states: f[0] ... f[1]', and exact transition equations. The manuscript never states who fills these fields. If these fields are filled by human authors knowing the solution, then the reported improvement from 5 to 46 AC (Section 6, Figure 19) demonstrates the model's ability to implement a provided algorithm specification, not an improvement in competitive-program generation ability. This confound must be resolved by stating the provenance of the prompt fields and, ideally, by ablating Phase 3 with model-generated or generic information.
  2. [§2.2, §4.1] The evaluation uses temperature=0.7 and exactly one submission per problem. Tables 3 and 4 and the Sankey diagram report point estimates (5/80 AC, 63 WA, 5 TLE, 7 CE) without repeated runs, pass@k estimates, or confidence intervals. Because decoding is stochastic, the headline numbers are single draws; a small number of borderline problems flipping could change the 5/80 or 46/80 values materially. The paper should report variance over repeated sampling or at least pass@k estimates, and it should soften claims that are stated as exact percentages.
  3. [§2.1.2, Table 1] The four filtering criteria remove 21 'golden' problems, 5 geometry-with-images problems, 9 problems with difficult-to-process descriptions, and 2 non-submittable problems, so the remaining 80-problem benchmark is a deliberately easier subset of real contests. Claims in the abstract and introduction about LLM performance on 'real-world competitive programming' should be scoped to this curated benchmark. The paper should state explicitly that the 5/80 result is likely an upper bound on performance over the full contest pool (because the removed problems are the hardest), or it should report results on the unfiltered pool where submission is possible. As written, neither the evaluation nor the improvement result is evidence about uncurated 2024 ICPC/CCPC contests.
  4. [§5 and §6] No baseline or ablation isolates the contribution of the framework components. The improvement is reported only as the combined effect of Phase 1 error diagnosis, Phase 2 multi-turn dialogue repair, and Phase 3 information-augmented regeneration. Without comparisons against (a) simple re-generation with the same basic prompt, (b) repair using only the OJ error status without the taxonomy, or (c) Phase 2 alone versus Phase 3 alone, the 'taxonomy-driven' aspect is not validated, and the 46/80 number could partly reflect multiple attempts and additional prompt information rather than the taxonomy. At minimum, the authors should add an ablation or clearly label the result as an overall pipeline measurement in a human-in-the-loop setting.
  5. [§4.2] The taxonomy validation procedure is not independent: the paper states that for the remaining 20% of incorrect programs, 'If a previously unobserved error type emerged during this step, we added new categories accordingly.' This means the validation set was used to extend the taxonomy before the final kappa was computed, so the reported Cohen's kappa of 0.86 cannot be interpreted as an out-of-sample reliability estimate of a fixed taxonomy. The authors should either fix the taxonomy before validation and report kappa on that fixed version, or explicitly describe the extension process and report agreement before and after extension.
minor comments (7)
  1. [§4.1, text before Table 4] The sentence '37 resulted in W A, 4 in TLE, and 37 were CE' is inconsistent with Table 4, which reports 4 CE for single-algorithm problems; the text should say '4 CE'.
  2. [Figure 5] The category 'Others (7.0%)' appears in the taxonomy figure but is never defined or explained in the text; please state what this category contains.
  3. [§2.1.1] The contest name 'CCPC-Jina' should be 'CCPC-Jinan' to match the standard romanization and the provided URL.
  4. [§5.3] The text refers to 'CCPC-FM, Problem F (Perfect Square)', but the earlier contest list uses the label CCPC-SF for the contest at the given URL; please correct the inconsistent contest abbreviation.
  5. [§5.2] The mapping between error types and the seven repair strategies is said to be provided only on GitHub; include at least a summary table of this mapping in the paper to make the framework self-contained.
  6. [Figure 19] The Sankey diagram contains the local file path 'C:/Users/86199/Downloads/sankeymatic_20250624_192952.svg' in the image; please regenerate the figure without the local path.
  7. [§7.2] The statement about DeepSeek's pre-training cutoff cites reference [29], which is a general survey on code generation; please cite the DeepSeek-R1 technical report or another primary source for the training-data cutoff claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RQ1 is an external OJ measurement and the improvement pipeline is empirical, with only a minor methodological self-citation.

full rationale

The RQ1 result (5/80 AC) is an independent empirical measurement: the basic prompt is fixed, DeepSeek-R1 is sampled once per problem, and correctness is judged by external OJ platforms, so no circular reduction is present. The RQ2 error taxonomy is descriptive: it is built by manual annotation with Cohen's kappa 0.86 and a held-out validation set, and it is not used as both the input and the output of a prediction. The improvement section is an engineering pipeline; Phase 3 explicitly discloses that structured scaffolding (Problem Modeling, Algorithm Ideas, Implementation Details, Constraints) is injected into the prompt, and the example in Figure 18 contains the solution's recurrence relations. This is a serious validity limitation for any claim of autonomous LLM improvement, but it is not circularity in the strict sense: the scaffolded prompt is an input and the generated code is an output, not an equation that reduces to its input by construction. The only self-citation ([20], used for the open coding procedure) is methodological, non-load-bearing, and supported by the paper's own inter-annotator agreement procedure. Overall, no claim in the paper is equivalent to its inputs by definition.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on the composition of the curated benchmark, the assumption that the 2024 contests are outside DeepSeek-R1's training data, and the assumption that human-authored solution hints in Phase 3 are a fair component of an LLM improvement framework. The temperature and filtering rules are hand-set choices that directly influence the reported numbers.

free parameters (4)
  • DeepSeek-R1 sampling temperature = 0.7
    Chosen in Section 2.2 based on prior work on adaptive temperature; affects generation diversity and therefore AC counts. No sensitivity analysis is reported.
  • Benchmark filtering criteria = Filters 1-4 (gold, geometry, long description, non-submittable)
    Section 2.1.2 defines four hand-designed rules that reduce 117 contest problems to 80; the composition of the benchmark is determined by these choices, and it directly drives the 5/80 baseline.
  • Development/validation split for taxonomy = 80/20 (60 and 15 programs)
    Section 4.2 randomly splits 75 non-AC programs; error rate percentages depend on this split.
  • Repair strategy mapping = Seven strategies mapped to error types
    Section 5.2 selects repair prompts based on the taxonomy; the mapping is hand-authored and affects which failures Phase 2 can fix.
assumptions (4)
  • domain assumption Online judge verdicts from VJudge and Codeforces are correct ground truth for problem solutions.
    Section 4.1 relies on OJ statuses (AC/WA/TLE/CE) as the sole correctness oracle; hidden test data quality is assumed sufficient.
  • domain assumption The 2024 ICPC/CCPC contests postdate DeepSeek-R1's training data, mitigating data leakage.
    Section 7.2 asserts this using reference [29], a survey that does not provide the DeepSeek-R1 training cutoff; the claim is not directly evidenced in the paper.
  • ad hoc to paper The filtering criteria do not remove a disproportionate share of problems where LLMs would fail, so the curated 80 reflect general LLM capability.
    Section 2.1.2 explicitly filters to align with current LLM capability, which biases the benchmark toward easier problems; the external validity claim depends on this assumption being false or harmless.
  • ad hoc to paper Human-written algorithm ideas in the Phase 3 prompt template encode correct solution strategies.
    Figure 18 shows the template includes Problem Modeling, Algorithm goals, Algorithm ideas, and Implementation details; the 46/80 result assumes this scaffolding is a legitimate evaluation of LLM improvement rather than human problem-solving.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating and Improving Large Language Models for Competitive Program Generation." pith.science (2026). https://pith.science/paper/VZWNL7GY

@misc{pith2026250622954,
  author       = {Pith},
  title        = {Pith review of: Evaluating and Improving Large Language Models for Competitive Program Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZWNL7GY}},
  note         = {Machine review of arXiv:2506.22954}
}
read the original abstract

Context: Due to the demand for strong algorithmic reasoning, complex logic implementation, and strict adherence to input/output formats and resource constraints, competitive programming generation by large language models (LLMs) is considered the most challenging problem in current LLM-based code generation. However, previous studies often evaluate LLMs using simple prompts and benchmark datasets prone to data leakage. Moreover, prior work has limited consideration of the diversity in algorithm types and difficulty levels. Objective: In this study, we aim to evaluate and improve LLMs in solving real-world competitive programming problems. Methods: We initially collect 117 problems from nine regional ICPC/CCPC contests held in 2024 and design four filtering criteria to construct a curated benchmark consisting of 80 problems. Leveraging DeepSeek-R1 as the LLM, we evaluate its competitive program generation capabilities through the online judge (OJ) platforms, guided by a carefully designed basic prompt. For incorrect submissions, we construct a fine-grained error taxonomy and then propose a targeted improvement framework by combining a multi-turn dialogue-based repair phase and an information-augmented regeneration phase. Results: Experimental results show that only 5 out of 80 problems are fully accepted when using basic prompts. For the unsolved problems, we construct the error taxonomy, including general errors (such as design, boundary, condition, data type, syntax, and input/output errors) and specialized errors (such as those in mathematical problems, greedy algorithms, and graph theories). After applying our proposed improvement strategies, we substantially increased the number of correct solutions, with 46 out of 80 problems successfully accepted.

Figures

Figures reproduced from arXiv: 2506.22954 by the authors.

Figure 1
Figure 1. Our empirical study methodology for evaluating and improving LLM-based competitive program generation [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. statistics of different problem types T1 Basic Algorithms (BA). Basic algorithms are widely applicable across various problems and are typically fundamen￾tal, concise, and conceptually straightforward. • T1.1 Simulation (SI). Simulates specific processes or operations described in the problem. • T1.2 Construction (CO). Builds outputs directly by ex￾ploiting problem patterns or structures. 18https://oi-wiki.org/ 19ht… view at source ↗
Figure 5
Figure 5. Hierarchical error taxonomy of LLM-generated competitive programs. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (13 more)
Figure 8
Figure 8. Figure 8: An example of GE1.3 error • Inappropriate Data Structure Selection (GE1.4). This type of error arises when the selected data structure does not align with the operational requirements or constraints of the problem. Such a mismatch can lead to low time or space efficien…
Figure 7
Figure 7. Figure 7: An example of GE1.2 error • Overly Complex or Inefficient Design (GE1.3). This type of error refers to situations where the solver con￾structs a solution that, while logically correct, incorpo￾rates unnecessary layers of computation, overly complex control flow, or ine…
Figure 9
Figure 9. Figure 9: An example of GE2.1 error The 2024 ICPC Kunming Invitational Contest Problem A Two-star Contest ERROR : It was not checked whether the filled value was ≤ k [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: An example of GE2.2 error Condition-related Errors (GE3). Faulty boolean expres￾sions (such as incorrect operators, misinterpreted precedence, or misplaced negations) can misdirect control flow or prema￾turely terminate loops. These errors typically result in partiall…
Figure 12
Figure 12. Figure 12: An example of GE6.2 error include misusing modular arithmetic identities, applying combinatorial formulas without handling edge conditions, or invoking theorems under invalid assumptions. • Special Mathematical Structure Handle Errors (AE1.2). When dealing with data w…
Figure 13
Figure 13. Figure 13: An example of AE4.1 error • Over-pruning or Missing Transitions (AE6.2). Search algorithms often include pruning to improve efficiency, but overly aggressive or incorrect pruning can remove valid paths. Similarly, neglecting certain transitions in the state graph resu…
Figure 14
Figure 14. Figure 14: Improvement framework for LLM-based competitive program generation [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: A structured workflow for identifying and diagnosing errors in non-AC programs. [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Basic prompt of 2024 China Collegiate Programming Contest [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: A multi-turn dialogue-based repair strategy example. [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: A prompt template example for CCPC-FM, Problem F. [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 20
Figure 20. Figure 20: Comparison of program generation results before and after improvement in terms of di [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]
Figure 21
Figure 21. Figure 21: Comparison of program generation results before and after improve [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Independent Sampling Outperforms Agentic Reasoning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    On Codeforces problems, independent k-shot sampling achieves better accuracy-cost and accuracy-query tradeoffs than agentic reasoning, even with prompt caching.

Reference graph

Works this paper leans on

37 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jiang, F

    J. Jiang, F. Wang, J. Shen, S. Kim, S. Kim, A survey on large language models for code generation, arXiv preprint arXiv:2406.00515 (2024)

    arXiv 2024

  2. [2]

    Zhang, D

    J. Zhang, D. Li, J. C. Kolesar, H. Shi, R. Piskac, Automated feed- back generation for competition-level code, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineer- ing, 2022, pp. 1–13

    work page 2022

  3. [3]

    M. S. Hossain, A. Tabassum, M. F. Arefin, T. S. Zaman, Llm-pros: Ana- lyzing large language models’ performance in competitive problem solv- ing, arXiv preprint arXiv:2502.04355 (2025)

    work page arXiv 2025

  4. [4]

    S. Li, F. Yang, J. Liu, Y . Li, Z. Liu, H. Sun, Y . Wang, Y . Chen, Y . Fu, W. Shi, Z. Liu, W. Tang, Y . Shen, B. Tang, M. Ding, J. Tang, J. Song, Competition-level code generation with alphacontest, arXiv preprint arXiv:2312.05820 (2023)

    work page Pith review arXiv 2023

  5. [5]

    Y . Lu, Y . Zhang, G. Li, S. Huang, Z. Liu, Y . Chen, X. Han, Y . Xu, F. Wei, Magicoder: The rise of generative ai for code understanding and genera- tion, arXiv preprint arXiv:2403.05530 (2024)

    arXiv 2024

  6. [6]

    T. Wang, N. Zhou, Z. Chen, Enhancing computer programming educa- tion with llms: A study on e ffective prompt engineering for python code generation, arXiv preprint arXiv:2407.05437 (2024)

    work page Pith review arXiv 2024

  7. [7]

    Ridnik, D

    T. Ridnik, D. Kredo, I. Friedman, Code generation with alpha- codium: From prompt engineering to flow engineering, arXiv preprint arXiv:2401.08500 (2024)

    arXiv 2024

  8. [8]

    Z. Wang, M. Shao, J. Bhandari, L. Mankali, R. Karri, O. Sinanoglu, M. Shafique, J. Knechtel, Vericontaminated: Assessing llm-driven verilog coding for data contamination, arXiv preprint arXiv:2503.13572 (2025)

    arXiv 2025

Show all 37 references
  1. [9]

    Matton, T

    A. Matton, T. Sherborne, D. Aumiller, E. Tommasone, M. Alizadeh, J. He, R. Ma, M. V oisin, E. Gilsenan-McMahon, M. Gall ´e, On leakage of code generation evaluation datasets, arXiv preprint arXiv:2407.07565 (2024)

    2024 arXiv

  2. [10]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Ed- wards, Y . Burda, N. Joseph, G. Brockman, et al., Evaluating large lan- guage models trained on code, arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  3. [11]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al., Program synthesis with large language models, arXiv preprint arXiv:2108.07732 (2021)

    2021 arXiv

  4. [12]

    Riddell, A

    M. Riddell, A. Ni, A. Cohan, Quantifying contamination in evaluat- ing code generation capabilities of language models, arXiv preprint arXiv:2403.04811 (2024)

    2024 arXiv

  5. [13]

    Huang, Z

    Y . Huang, Z. Lin, X. Liu, Y . Gong, S. Lu, F. Lei, Y . Liang, Y . Shen, C. Lin, N. Duan, et al., Competition-level problems are e ffective llm evaluators, arXiv preprint arXiv:2312.02143 (2023)

    2023 arXiv

  6. [14]

    Zheng, Z

    Z. Zheng, Z. Cheng, Z. Shen, S. Zhou, K. Liu, H. He, D. Li, S. Wei, H. Hao, J. Yao, et al., Livecodebench pro: How do olympiad medalists judge llms in competitive programming?, arXiv preprint arXiv:2506.11928 (2025)

    2025 arXiv

  7. [15]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  8. [16]

    Y . Zhu, J. Li, G. Li, Y . Zhao, Z. Jin, H. Mei, Hot or cold? adaptive temperature sampling for code generation with large language models, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 38, 2024, pp. 437–445

    2024

  9. [17]

    Z. Liu, Y . Tang, X. Luo, Y . Zhou, L. F. Zhang, No need to lift a fin- ger anymore? assessing the quality of code generation by chatgpt, IEEE Transactions on Software Engineering (2024) 1–35

    2024

  10. [18]

    C. Liu, X. Bao, H. Zhang, N. Zhang, H. Hu, X. Zhang, M. Yan, Guiding chatgpt for better code generation: An empirical study, in: 2024 IEEE International Conference on Software Analysis, Evolution and Reengi- neering (SANER), IEEE, 2024, pp. 102–113

    2024

  11. [19]

    J. Wen, Z. Chen, Y . Liu, Y . Lou, Y . Ma, G. Huang, X. Jin, X. Liu, An empirical study on challenges of application development in serverless computing, in: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations ...

    2021

  12. [20]

    X. Chen, C. Gao, C. Chen, G. Zhang, Y . Liu, An empirical study on chal- lenges for llm application developers, ACM Transactions on Software En- gineering and Methodology 1 (1) (2025) 1–35. doi:10.1145/3715007

    2025 doi

  13. [21]

    C. B. Seaman, Qualitative methods in empirical studies of software engi- neering, IEEE Transactions on software engineering 25 (4) (1999) 557– 572

    1999

  14. [22]

    Cohen, A coe fficient of agreement for nominal scales, Educational and psychological measurement 20 (1) (1960) 37–46

    J. Cohen, A coe fficient of agreement for nominal scales, Educational and psychological measurement 20 (1) (1960) 37–46

    1960

  15. [23]

    J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data, biometrics (1977) 159–174

    1977

  16. [24]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: How language models use long contexts, arXiv preprint arXiv:2307.03172 (2023)

    2023 arXiv

  17. [25]

    Reynolds, K

    L. Reynolds, K. McDonell, Prompt programming for large language mod- els: Beyond the few-shot paradigm, in: Extended abstracts of the 2021 CHI conference on human factors in computing systems, 2021, pp. 1–7

    2021

  18. [26]

    Beurer-Kellner, M

    L. Beurer-Kellner, M. Fischer, M. Vechev, Prompting is programming: A query language for large language models, Proceedings of the ACM on Programming Languages 7 (PLDI) (2023) 1946–1969

    2023

  19. [27]

    B. Chen, Z. Zhang, N. Langren ´e, S. Zhu, Unleashing the potential of prompt engineering in large language models: a comprehensive review, arXiv preprint arXiv:2310.14735 (2023)

    2023 arXiv

  20. [28]

    Y . Yan, S. Wang, J. Huo, H. Li, B. Li, J. Su, X. Gao, Y .-F. Zhang, T. Xu, 18 Z. Chu, et al., Errorradar: Benchmarking complex mathematical rea- soning of multimodal large language models via error detection, arXiv preprint arXiv:2410.04509 (2024)

    2024 arXiv

  21. [29]

    Zhang, K

    H. Zhang, K. Zhang, Z. Li, J. Li, J. Li, Y . Li, Y . Zhao, Y . Zhu, F. Liu, G. Li, et al., Deep learning for code generation: a survey, Science China Information Sciences 67 (9) (2024) 191101

    2024

  22. [30]

    Shakya, F

    R. Shakya, F. Vadiee, M. Khalil, A showdown of chatgpt vs deepseek in solving programming tasks, in: 2025 International Conference on New Trends in Computing Sciences (ICTCS), IEEE, 2025, pp. 413–418

    2025

  23. [31]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al., Measuring coding challenge competence with apps, arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  24. [32]

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Ec- cles, J. Keeling, F. Gimeno, A. Dal Lago, et al., Competition-level code generation with alphacode, Science 378 (6624) (2022) 1092–1097

    2022

  25. [33]

    Zhang, Z

    K. Zhang, Z. Li, J. Li, G. Li, Z. Jin, Self-edit: Fault-aware code editor for code generation, in: Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (V olume 1: Long Papers), 2023, pp. 769–787

    2023

  26. [34]

    M. A. Islam, M. E. Ali, M. R. Parvez, Mapcoder: Multi-agent code gener- ation for competitive problem solving, arXiv preprint arXiv:2405.11403 (2024)

    2024 arXiv

  27. [35]

    Souza, R

    D. Souza, R. Gheyi, L. Albuquerque, G. Soares, M. Ribeiro, Code gen- eration with small language models: A deep evaluation on codeforces, arXiv preprint arXiv:2504.07343 (2025)

    2025

  28. [36]

    L. Yang, R. Jin, L. Shi, J. Peng, Y . Chen, D. Xiong, Probench: Bench- marking large language models in competitive programming, arXiv preprint arXiv:2502.20868 (2025). Minnan Wei is currently pursuing a Master’s degree at the School of Artificial Intelligence and Computer Sci...

    2025 arXiv

  29. [2023]

    More information can be found at: https://xchencs.github.io/index.html

    He is the editorial board member of Information and Software Technology. More information can be found at: https://xchencs.github.io/index.html. Menglin Zheng is currently pursuing a Bachelor’s degree in Software Engineering at the School of Artificial Intelligence and Compute...

    1976

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.