Pith. sign in

REVIEW 3 major objections 6 minor 24 references

ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark of 118 ICPC contest problems separates top AI reasoning models that other coding tests do not, and the paper's Refine@K metric shows that even the best model solves only 28.8% of them, far below human medalists.

desk verdict Useful and reproducible benchmark, but the 'zero false positives' claim for LLM-generated tests is not supported by the validation and needs a proper false-positive analysis before rankings are trusted. read the letter →

arxiv 2506.04894 v1 pith:XEUSYR63 submitted 2025-06-05 cs.CL

Shiyi Xu , Yiwen Hu , Yingqian Min , Zhipeng Chen , Wayne Xin Zhao , Ji-Rong Wen This is my paper
classification cs.CL
keywords ICPC-EvalcompetitiveprogrammingbenchmarkRefine@Kreasoningmodelstest-timescalingexecutionfeedbacktestcasegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ICPC-Eval, a benchmark of 118 problems drawn from 11 recent ICPC contests, built to be hard enough that state-of-the-art language models cannot saturate it. To make the benchmark usable offline, the authors generate and validate C++ input generators using language models, and they propose Refine@K, a metric that scores whether a model can fix its own code within K attempts given execution feedback. On the benchmark, top reasoning models solve only a fraction of problems—best is o3-mini High at 28.8% Refine@5—far below human medalists, and models that think longer score higher. The paper argues that Refine@K captures iterative self-correction ability that Pass@K misses, making it a better test-time scaling metric for reasoning models.

What carries the argument

Two mechanisms carry the argument. The Refine@K protocol gives a model K chances: it first writes a solution, then receives compiled error messages, mismatched example outputs, or hidden-test error types (WA, TLE, RE, MLE), and revises its code accordingly. The test-case pipeline uses LLM-generated random and corner-case input generators, with outputs produced by an accepted solution, validated so that three hand-collected wrong programs (WA, TLE, RE) fail and the accepted program passes; the paper calls this 'zero false positives'. Refine@K makes the evaluation a test of response to feedback, while the validated local test suite makes offline evaluation feasible.

What would settle it

Take the set of all official-submitted programs for these 118 problems (or a random sample) with known verdicts on the contest judges, run them against ICPC-Eval's generated test suite, and count mismatches: if a substantial fraction of official WA/TLE programs pass the generated tests, or official AC programs fail them, the zero-false-positive claim and the reported solve rates would not transfer to real contest performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that ICPC-Eval establishes a notably more challenging baseline for state-of-the-art models than existing coding benchmarks, and that Refine@K measures a capability—iterative repair from execution feedback—that Pass@K ignores. Concretely, on 118 recent ICPC problems, o3-mini High reaches 28.8% solved at Refine@5, Gemini 2.5 Pro Exp 22.0%, DeepSeek-R1 14.4%, and non-reasoning models stay below 5%; the same models score far higher on LiveCodeBench (e.g., o3-mini High 67.4%). Ablation studies on same-base-model pairs (QwQ-32B vs Qwen2.5-Coder-32B; DeepSeek-R1 vs DeepSeek-V3) show that Refine@K consistently exceeds Pass@K for reasoning models while the reverse holds for non-reasoning models, indicating that the refinement advantage is attributable to reasoning, not to resampling.

Load-bearing premise

The load-bearing assumption is that the LLM-generated local test cases are faithful stand-ins for the official hidden tests: any correct solution passes them and any incorrect solution fails them, something the paper verifies with just three hand-picked wrong programs and one accepted solution per problem.

Editorial extensions

If this is right

  • ICPC-Eval separates top reasoning models that cluster together on LiveCodeBench, e.g., Grok 3 Mini Beta 66.7% vs o3-mini High 67.4% there, but 11.8% vs 28.8% here.
  • Refine@5 correlates with average output length, suggesting the metric tracks the test-time compute spent on deeper chains of thought.
  • Reasoning models gain from multi-turn feedback while non-reasoning models are often hurt by it, so Refine@K rewrites conclusions drawn from Pass@K about model capability gaps.
  • Current models remain far from human-level performance on ICPC-level problem solving, since even the best model solves under one third of the problems that human medallists solve.
  • The benchmark can be refreshed annually with new ICPC contests to resist data contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the generated test cases are not distributed like official hidden tests, model rankings on ICPC-Eval may shift; the paper's zero-false-positive claim rests on a small validation set, so published solve rates are best read as approximate lower bounds rather than exact contest scores.
  • Refine@K could be adopted as a general protocol for evaluating agentic coding, and combined with budgets on both attempts and tokens to separate 'can repair' from 'can solve from scratch'.
  • The refinement penalty for non-reasoning models suggests a testable prediction: fine-tuning on execution feedback would narrow the Refine@K gap more than additional supervised pretraining on code.
  • Because the benchmark includes only 118 problems with heavy per-problem annotation effort, scaling it up will require automating special-judge construction and validation, which is currently the least automated part.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces ICPC-Eval, a competitive-programming benchmark of 118 problems from recent ICPC World Finals, Continental Finals, and Regional contests. Each problem is paired with an LLM-generated local test suite and, for 12 problems, a custom special judge. The paper also proposes Refine@K, a metric that counts whether a model can produce a correct solution within K turns when given execution feedback (compilation errors, example-test mismatches, or hidden-test error types). The authors evaluate 15 state-of-the-art LLMs, report that the best model (o3-mini High) reaches 28.8% Refine@5, and claim that Refine@K better exposes the reflective capabilities of reasoning models than Pass@K. A paired-model ablation (QwQ vs Qwen2.5-Coder and DeepSeek-R1 vs DeepSeek-V3) supports the view that reasoning models, unlike non-reasoning models, improve with iterative feedback.

Significance. If the local test suites are faithful to the official hidden tests, ICPC-Eval addresses a real gap: it provides a high-difficulty, locally executable, contamination-controlled coding benchmark with a metric that captures iterative repair—a capability that independent sampling (Pass@K) misses. The 118-problem set is larger than most high-difficulty collections and spans a realistic distribution of algorithmic topics and contest difficulty. The paired-model ablations are well designed and support the specific claim that reasoning models benefit from execution feedback, while non-reasoning models do not. The public release of the test-generation code, local evaluation toolkit, and special judges is a practical contribution that should facilitate reproducibility and future benchmarking. The main risk to significance is the validation of the generated test cases, which is currently too weak to justify the strong 'zero false positives' claim and, by extension, the reported absolute solve rates and rankings.

major comments (3)
  1. [3.3] The 'zero false positives' claim is not established. The validation procedure uses only one accepted program and three known incorrect programs (one Wrong Answer, one Time Limit Exceeded, one Runtime Error) per problem. A set of three programs cannot cover the space of plausible incorrect solutions; an incorrect program that passes the LLM-generated cases but fails the official tests would be scored as correct, directly inflating Refine@K and distorting model rankings. Since every headline number in Tables 3 and 4 and Figure 1 flows through these local test suites, this is a load-bearing assumption. Please either temper the 'zero false positives' wording to reflect the actual evidence, or validate on a substantially larger and more varied set of wrong solutions (e.g., submissions from the online judge with various error statuses, or systematically mutated versions of accepted solutions) and report the resulting false-positive and false-negative rates per problem.
  2. [3.3] The validation procedure also does not address false negatives: a correct solution might fail the generated tests if the generator emits inputs that violate stated constraints or if the single accepted solution used as an oracle is incorrect or nondeterministic on those inputs. The paper relies on one accepted program per problem and does not report any check that the generated inputs satisfy the problem constraints (beyond the prompt instruction to the generator). Without a systematic constraint validator or multiple accepted solutions, the benchmark may mark correct solutions as wrong, deflating scores. Please add a post-generation constraint check (or a second independent solution as oracle) and report, per problem, how many generated cases were used, how many were discarded, and how the oracle was verified.
  3. [5.2] The claim that Refine@K 'serves as a more effective test-time estimator than Pass@K' (Section 3.4, validated in Section 5.2) is only partially supported by the current experiments. The paired-model comparisons in Figure 4 show that reasoning models improve more under Refine@K than under Pass@K, but the two metrics are compared at different levels of total computation: Refine@K consumes multiple rounds of generation plus feedback, whereas Pass@K uses K independent samples with no feedback. The observed gap could therefore reflect a difference in total compute or in the amount of information provided, rather than a difference in 'intrinsic reasoning ability.' The paper should either (a) provide a cost-matched comparison (e.g., matching total token budget or wall-clock time between Refine@K and an equivalent number of independent samples) or (b) explicitly reframe Refine@K as measuring a different capability—iterative repair from execution feedback—rather than as a 'more effective estimator' of the same capability measured by Pass@K.
minor comments (6)
  1. [3.3] The manuscript does not state the number of generated test cases per problem (both random and corner) or the criteria for selecting that number. Please report these counts, as they directly affect the granularity of the local evaluation and its sensitivity to rare failure modes.
  2. [5.2 (Figure 4)] The legend entries 'Refine@5, Pass@4' are confusing: it is not clear whether the plotted curves correspond to Refine@5 and Pass@4 as separate quantities or to some combined evaluation. Please clarify what is plotted for each model and attempt value.
  3. [4.1] The evaluation uses temperature 0.6 and top_p 0.95 for locally hosted models but default hyperparameters for API-based models. This difference may confound cross-model comparisons (e.g., o3-mini High vs. locally hosted QwQ-32B). Please discuss whether the main results are sensitive to this choice.
  4. [1 / References] The text cites '[8]' as 'LeetCode-Hard', but Reference [8] is the Reflexion paper; either the citation is incorrect or the related-work description of LeetCode-Hard needs its own reference. Please check all citation-number pairings.
  5. [Table 1] The row for 'ProBench' is listed with source 'ICPC' but ProBench is not discussed in the text. Please add a brief description of ProBench or remove the row to avoid confusion.
  6. [3.3] The sentence 'we ask the model to regenerate the generators that failed in any these check to ensuring zero false positives' contains typos ('any these check', 'to ensuring'). Please proofread this passage for grammar and clarity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark and Refine@K metric are externally anchored; residual concerns are validity risks, not circular reductions.

full rationale

The central evaluation quantities are fixed by external sources rather than by the paper's own fitted constants: the problems come from real ICPC contests, the oracle outputs come from accepted solutions on QOJ, and model scores are obtained by running generated code against local tests. The test-case generators are produced by an LLM, but the paper checks them against known accepted and known incorrect programs; this is an external validation step, not a definitional equivalence. Refine@K is defined as a pass-within-K-feedback protocol and is then compared empirically against Pass@K on matched base-model pairs; no equation makes the stated advantage of Refine@K true by construction. The paper's self-citations—the authors' earlier survey and their STILL-3-Tool-32B model used as one of fifteen evaluated systems—are not load-bearing premises for the benchmark's conclusions. The 'zero false positives' claim rests on a small validation sample and is therefore a validity risk, but that is a correctness concern rather than circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on external contest problems and external accepted solutions, so the core is grounded. The fragile parts are the assumptions about test-case fidelity and contamination, which are plausible but not proven. No invented physical or mathematical entities are introduced.

free parameters (2)
  • Refine@K budget K = 5 for primary metric, 20 in ablations
    Chosen by the authors as the primary evaluation budget; all reported scores depend on this choice.
  • Number of generated test cases per problem = Not specified in the paper
    The size and composition of the local test suite are not reported, which affects the stringency and false-positive behavior of the benchmark.
assumptions (4)
  • domain assumption QOJ accepted solutions used to generate expected outputs are correct.
    The benchmark's ground truth outputs are derived from these programs; if any is actually incorrect, the generated expected outputs are wrong. Invoked in Section 3.3.
  • ad hoc to paper Validation against three known incorrect programs (WA, TLE, RE) suffices to guarantee zero false positives.
    This is the paper's explicit standard (Section 3.3), but it is logically insufficient; false positives could still exist for other wrong programs.
  • domain assumption Recent contests (Oct-Dec 2024 plus 2023 WF) and VJudge anti-crawling ensure minimal data contamination.
    The benchmark's value as a future-proof eval depends on models not having memorized these problems; no direct leakage test is provided. Section 3.1.
  • domain assumption LLM-generated generators produce inputs representative of official hidden test cases.
    Random and corner-case generators are prompted from the problem statement; there is no evidence that they cover the difficulty distribution of official tests. Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests." pith.science (2026). https://pith.science/paper/XEUSYR63

@misc{pith2026250604894,
  author       = {Pith},
  title        = {Pith review of: ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEUSYR63}},
  note         = {Machine review of arXiv:2506.04894}
}
read the original abstract

With the significant progress of large reasoning models in complex coding and reasoning tasks, existing benchmarks, like LiveCodeBench and CodeElo, are insufficient to evaluate the coding capabilities of large language models (LLMs) in real competition environments. Moreover, current evaluation metrics such as Pass@K fail to capture the reflective abilities of reasoning models. To address these challenges, we propose \textbf{ICPC-Eval}, a top-level competitive coding benchmark designed to probing the frontiers of LLM reasoning. ICPC-Eval includes 118 carefully curated problems from 11 recent ICPC contests held in various regions of the world, offering three key contributions: 1) A challenging realistic ICPC competition scenario, featuring a problem type and difficulty distribution consistent with actual contests. 2) A robust test case generation method and a corresponding local evaluation toolkit, enabling efficient and accurate local evaluation. 3) An effective test-time scaling evaluation metric, Refine@K, which allows iterative repair of solutions based on execution feedback. The results underscore the significant challenge in evaluating complex reasoning abilities: top-tier reasoning models like DeepSeek-R1 often rely on multi-turn code feedback to fully unlock their in-context reasoning potential when compared to non-reasoning counterparts. Furthermore, despite recent advancements in code generation, these models still lag behind top-performing human teams. We release the benchmark at: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs

Figures

Figures reproduced from arXiv: 2506.04894 by the authors.

Figure 1
Figure 1. Average number of problems solved per contest (typically 12 problems) by AI models [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The complete pipeline for test case generation and validation, enabling efficient local [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Comparison of Refine@K and Pass@K methods across different models. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 canonical work pages

  1. [1]

    A Survey of Large Language Models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A Survey of Large Language Models. (arXiv:2303.18223), 2025

    arXiv 2025

  2. [2]

    https://openai.com/o1/, 2024

    Introducing OpenAI o1. https://openai.com/o1/, 2024

    work page 2024

  3. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  4. [4]

    Gemini 2.5: Our most intelligent AI model

    work page

  5. [5]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. (arXiv:2403.07974), 2024

    arXiv 2024

  6. [6]

    Can Language Models Solve Olympiad Programming? (arXiv:2404.10952), 2024

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can Language Models Solve Olympiad Programming? (arXiv:2404.10952), 2024

    arXiv 2024

  7. [7]

    CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings. (arXiv:2501.01257), 2025

    arXiv 2025

  8. [8]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. (arXiv:2303.11366), 2023

    arXiv 2023

Show all 24 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

  2. [10]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021. 11

    2021 arXiv

  3. [11]

    Measuring Coding Challenge Competence With APPS

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring Coding Challenge Competence With APPS. (arXiv:2105.09938), November 2021

    2021 arXiv

  4. [12]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022

  5. [13]

    Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty

    Mohammad Abdullah Matin Khan, M. Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval. (arXiv:2303.03004), November 2023

    2023 arXiv

  6. [14]

    Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. (arXiv:2401.01335), June 2024

    2024 arXiv

  7. [15]

    Multi-turn rl training for cuda kernel generation

    Carlo Baronio, Pietro Marsella, Ben Pan, and Silas Alberti. Multi-turn rl training for cuda kernel generation. https://cognition.ai/blog/kevin-32b

  8. [16]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Is Self-Repair a Silver Bullet for Code Generation? (arXiv:2306.09896), February 2024

    2024 arXiv

  9. [17]

    OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement. (arXiv:2402.14658), January 2025

    2025 arXiv

  10. [18]

    SPoC: Search-based Pseudocode to Code

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. SPoC: Search-based Pseudocode to Code

  11. [19]

    Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learning

    Haiming Wang, Mert Unsal, Xiaohan Lin, Mantas Baksys, Junqi Liu, Marco Dos Santos, Flood Sung, Marina Vinyes, Zhenzhe Ying, Zekai Zhu, Jianqiao Lu, Hugues de Saxcé, Bolton Bailey, Chendong Song, Chenjun Xiao, Dehao Zhang, Ebony Zhang, Frederick Pu, Han Zhu, Jiawei Liu, Jonas B...

    2025 arXiv

  12. [20]

    Grok 3 Beta — The Age of Reasoning Agents | xAI

  13. [21]

    QwQ-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. QwQ-32b: Embracing the power of reinforcement learning, March 2025

    2025

  14. [22]

    An empirical study on eliciting and improving r1-like reasoning models.arXiv preprint arXiv:2503.04548, 2025

    Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving r1-like reasoning models.arXiv preprint arXiv:2503.04548, 2025

    2025 arXiv

  15. [23]

    Claude 3.7 Sonnet and Claude Code

  16. [24]

    Category1EnglishName

    Qwen Team. Qwen3: Think Deeper, Act Faster. 12 A Prompts A.1 Prompt Used for ICPC-Eval Initial generation: You are a coding expert. Given a competition-level coding problem, you need to write a C++ program (C++23) to solve it. Please consider the efficiency and time complexity...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.