Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming

T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frontier LLMs uncover errors in over 5 percent of full-score competitive programming submissions.

desk verdict UOJ-Bench adds real-data hacking and repair tasks from UOJ submissions, but the 5% full-score error claim lacks any check that the detections are genuine bugs. read the letter →

arxiv 2606.12864 v1 pith:QOBGJW5G submitted 2026-06-11 cs.SE cs.AI

Tingqiang Xu , Hangrui Zhou , Tianle Cai , Alex Gu , Kaifeng Lyu This is my paper
classification cs.SEcs.AI
keywords UOJ-Benchcodegenerationhackingrepaircompetitiveprogramminglargelanguagemodelstest-timescalingonlinejudges
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces UOJ-Bench to test LLMs not just on writing code but on spotting and fixing errors in human submissions from a real online judge. Under basic one-shot prompting, even top models miss errors in more than half of submissions already flagged as wrong by the judge. Test-time scaling lifts detection rates above 90 percent, yet the compute expense makes it impractical for broad use. The central result is that the strongest scaled models still surface mistakes in more than 5 percent of submissions that received full marks from the judge, across roughly 30 problems.

What carries the argument

UOJ-Bench, a benchmark of code generation, hacking, and repair tasks built from real UOJ submissions and scored by UOJ’s native judge.

What would settle it

Apply the same top models with test-time scaling to a new collection of UOJ problems and measure whether they still flag errors in more than 5 percent of full-score submissions.

Watch

Extended reading notes

Core claim

UOJ-Bench draws three tasks—code generation, code hacking, and code repair—directly from real submissions on the Universal Online Judge and scores them with the judge’s own infrastructure. One-shot evaluation shows models fail to identify errors in more than 50 percent of incorrect submissions. Test-time scaling raises success above 90 percent at substantial inference cost. The same scaled models nevertheless expose errors in over 5 percent of full-score submissions on approximately 30 problems, indicating frontier LLMs can supply signals beyond standard judging.

Load-bearing premise

The chosen UOJ problems and submissions fairly represent typical competitive-programming errors and the judge’s test cases carry no systematic bias.

Editorial extensions

If this is right

  • LLMs can act as a supplementary error-detection layer on top of existing online judges.
  • Practical deployment requires either cheaper scaling methods or selective application to high-value submissions.
  • Complementary signals appear even when models are far from perfect on every incorrect submission.
  • Educational tools could incorporate LLM hacking outputs to give students additional feedback beyond pass/fail results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Hybrid judge-plus-LLM systems might reduce false positives in competitive programming contests.
  • The cost-accuracy trade-off suggests focusing scaling on problems where human graders already disagree.
  • Extending the benchmark to other online judges could test whether the 5-percent signal generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces UOJ-Bench, a benchmark constructed from real submissions to the Universal Online Judge (UOJ) to evaluate LLMs on code generation, code hacking (error detection in incorrect submissions), and code repair tasks, all assessed via UOJ's native judging infrastructure. Key empirical findings include: one-shot prompting yields error detection in under 50% of UOJ-flagged incorrect submissions, test-time scaling raises this above 90% (at high cost), and the strongest scaled models detect potential errors in over 5% of full-score submissions across roughly 30 problems, suggesting LLMs can supply complementary signals to standard judges.

Significance. If the 5% detection rate in full-score submissions reflects genuine errors missed by UOJ test suites, the work would demonstrate a concrete educational role for LLMs in competitive programming beyond generation, namely automated identification of subtle bugs for human learners. The benchmark's use of authentic submissions and an established external judge strengthens ecological validity compared to synthetic datasets. Reproducible evaluation via UOJ infrastructure is a methodological strength that could support follow-on studies on LLM-assisted code review.

major comments (2)
  1. [Abstract / results paragraph on full-score submissions] Abstract and results on full-score submissions: the claim that best models 'can uncover errors in over 5% of full-score submissions across roughly 30 problems' is presented without any reported validation (manual review, construction of counterexample tests that the submissions fail, or inter-rater agreement) that the LLM-flagged issues are actual bugs rather than false positives. Because UOJ is used as ground truth for these full-score cases, the complementary-signal conclusion rests on unverified challenges to that ground truth.
  2. [Evaluation methodology (§3–4)] Evaluation methodology (throughout §3–4): no details are supplied on model versions, exact prompt templates, criteria for selecting the ~30 problems or the submission sets, or any statistical controls for the post-hoc 5% analysis. This absence directly affects reproducibility of the quantitative performance numbers and the selection-effect risk in the central 5% claim.
minor comments (1)
  1. [Results on test-time scaling] The description of test-time scaling costs would benefit from explicit reporting of token counts or wall-clock times per problem to allow readers to assess practicality.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments identify two areas where the manuscript can be strengthened for rigor and reproducibility. We address each below and will incorporate revisions.

read point-by-point responses

  1. Referee: [Abstract / results paragraph on full-score submissions] Abstract and results on full-score submissions: the claim that best models 'can uncover errors in over 5% of full-score submissions across roughly 30 problems' is presented without any reported validation (manual review, construction of counterexample tests that the submissions fail, or inter-rater agreement) that the LLM-flagged issues are actual bugs rather than false positives. Because UOJ is used as ground truth for these full-score cases, the complementary-signal conclusion rests on unverified challenges to that ground truth.

    Authors: We agree that the 5% figure is presented without direct validation that the flagged issues are genuine bugs missed by UOJ (e.g., no manual review or counterexample tests are reported). The manuscript frames the result as suggestive ('suggesting that frontier LLMs can already provide complementary signals') rather than conclusive. In revision we will (1) add explicit language in the abstract and results section stating that these are LLM-detected potential issues requiring further verification, (2) expand the limitations paragraph to discuss the absence of ground-truth validation for full-score cases, and (3) note that the complementary-signal interpretation remains provisional. We will also consider adding a small-scale manual spot-check in an appendix if space permits. revision: yes

  2. Referee: [Evaluation methodology (§3–4)] Evaluation methodology (throughout §3–4): no details are supplied on model versions, exact prompt templates, criteria for selecting the ~30 problems or the submission sets, or any statistical controls for the post-hoc 5% analysis. This absence directly affects reproducibility of the quantitative performance numbers and the selection-effect risk in the central 5% claim.

    Authors: We concur that the current text omits key methodological details required for reproducibility. In the revised manuscript we will add a dedicated subsection (or expand §3) that specifies: the exact model versions and checkpoints used, the full prompt templates (or repository links), the selection criteria for the ~30 problems (submission volume, topic diversity, and problem difficulty), the sampling procedure for submission sets, and any statistical controls or post-hoc adjustments applied to the 5% analysis. This will also mitigate concerns about selection effects. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: purely empirical benchmark evaluation

full rationale

The paper introduces UOJ-Bench as a dataset of real submissions from the external Universal Online Judge platform and reports direct LLM performance metrics on code generation, hacking, and repair tasks using UOJ's native judging as ground truth. No mathematical derivations, equations, fitted parameters presented as predictions, ansatzes, or uniqueness theorems appear in the abstract or described methodology. Central claims rest on empirical measurements against an independent external system rather than any self-referential construction or self-citation load-bearing step. This matches the default case of a self-contained empirical benchmark with no load-bearing reductions to inputs.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claims rest on the domain assumption that UOJ verdicts constitute accurate ground truth and that the chosen submissions reflect typical error patterns; no free parameters or invented entities are introduced.

assumptions (1)
  • domain assumption UOJ submissions and judging provide reliable ground truth for code correctness and error presence
    All three tasks and the reported performance numbers are defined relative to UOJ verdicts as the reference standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming." pith.science (2026). https://pith.science/paper/QOBGJW5G

@misc{pith2026260612864,
  author       = {Pith},
  title        = {Pith review of: Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOBGJW5G}},
  note         = {Machine review of arXiv:2606.12864}
}
read the original abstract

Despite strong performance in competitive programming, the role of Large Language Models (LLMs) in supporting human learning in the same setting remains largely unexplored. In this work, we introduce UOJ-Bench, a benchmark designed to evaluate not only the problem-solving ability of LLMs, but also their ability to identify errors in human-written code -- a crucial educational activity traditionally supported by running test cases over online judge systems. UOJ-Bench consists of three distinct tasks: code generation, code hacking, and code repair, all constructed from real-world code submissions on the Universal Online Judge (UOJ) and evaluated through UOJ's native judging infrastructure. Our results show that under one-shot evaluation, even the strongest models fail to identify errors in more than 50% of a set of submissions that have been found to be incorrect by UOJ users. While test-time scaling improves success rates to above 90%, the substantial computational costs incurred from model inference limit its practicality for large-scale deployment. Despite these limitations, we find that the best-performing models under test-time scaling can uncover errors in over 5% of full-score submissions across roughly 30 problems, suggesting that frontier LLMs can already provide complementary signals beyond standard judging systems.

Figures

Figures reproduced from arXiv: 2606.12864 by the authors.

Figure 1
Figure 1. Detailed statistics of the UOJ Bench dataset. WA, TLE, RE, and MLE denote Wrong Answer, Time Limit Exceeded, Runtime Error, and Memory Limit Exceeded verdicts, respectively. uncommon for a solution to pass all standard test cases while still being incorrect upon closer examination. In this paper, we refer to errors that can be exposed by standard test cases as overt errors, and those that cannot be exposed by standa… view at source ↗
Figure 2
Figure 2. Cost-performance analysis across three tasks: (a) Generation, (b) Hacking (Hard), and (c) Repair (Hard). The plots compare success rates (Pass@1) against the average inference cost per problem. For GPT-OSS-120B and Gemini-3-pro-preview, we additionally illustrate test-time scaling capability by plotting Pass@k curves (up to k = 20, 000 and k = 100, respectively) on randomly selected subsets (100 samples for Task 2, … view at source ↗
Figure 3
Figure 3. Cost-performance analysis for Zero-Day Hacking. GPT￾OSS-120B under test-time scaling is the most cost-effective. Zero-Day Hacking. Beyond the Easy and Hard levels, we further introduce an extended evaluation set for code hack￾ing, Zero-Day-Hacking-5K, which contains 5,060 sampled full-score submissions. Unlike the Hard set, where covert er￾rors have already been exposed by human-generated hacks, this set contains su… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distribution of Problems by Source and Difficulty [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Zero-Day Hacking success rates across problems. The chart displays the percentage of target submissions (originally deemed Accepted) for which the models successfully generated a valid counter-example. The significant variance indicates that certain problems possess le…
Figure 6
Figure 6. Figure 6: Performance Scaling with Increased Computational Budget (Direct Sampling vs. Agentic Interaction). The plots compare the cumulative success rates of GPT-OSS-120B on Hacking (Left) and Repair (Right) tasks over 10 turns. LLMs in a real-world pedagogical environment, we …
Figure 7
Figure 7. Figure 7: Temporal generalization analysis on Hard problems. The plots display the success rates of Gemini-3-Pro-Preview and GPT-OSS-120B across years. While Code Generation shows a performance drop on recent data (suggesting contamination), Code Hacking and Self-Repair remain s…
Figure 8
Figure 8. Figure 8: presents the performance of Gemini-3-pro-preview and GPT-OSS-120B under test-time scaling, where the horizontal axis denotes the number of samples k and the vertical axis reports cumulative success rates. The curves illustrate how performance improves as additional sam…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A target-blind agent-generated test suite with a consensus-and-validator certification chain found 906 verified accepted-but-buggy submissions on AtCoder and led all baselines on post-cutoff Codeforces problems.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jimenez, C

    URL https://openreview.net/forum?id=chfJJYC3 iL. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations,

  2. [2]

    Taco: Topics in algorithmic code generation dataset

    URL https://openreview.net/forum?id=VTF8yN QM66. Li, R., Fu, J., Zhang, B.-W., Huang, T., Sun, Z., Lyu, C., Liu, G., Jin, Z., and Li, G. TACO: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852, 2023. Li, Y ., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hube...

    work page arXiv 2023

  3. [3]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    doi: 10.1126/science.abq1158. URL https: //www.science.org/doi/abs/10.1126/science.abq1158. Lin, H. M. and Cheng, H.-T. Gemini achieves gold-medal level at the international collegiate programming contest world finals. Google DeepMind Blog, September 2025. URL https://deepmind.google/blog/gemini-achieves-gol d-medal-level-at-the-international-collegiate-p...

    work page doi:10.1126/science.abq1158 2025

  4. [4]

    These excluded types often require multi-file submissions or non-static hack protocols, which introduce ambiguity and incompatibility with current LLM benchmarking standards

    To ensure a rigorous and unified evaluation pipeline, we exclude non-standard formats because they deviate from the widely adopted Single-File, Standard I/O paradigm. These excluded types often require multi-file submissions or non-static hack protocols, which introduce ambiguity and incompatibility with current LLM benchmarking standards

  5. [5]

    We exclude problems heavily reliant on randomized algorithms (e.g., rolling hashes). Hacks for such problems often rely on brute-forcing collisions against specific modular constants rather than exploiting logical flaws, which does not align with our focus on reasoning

  6. [6]

    self-hacks

    We filter out "self-hacks" (users hacking their own code for recreation) and submissions containing leftover debugging artifacts. To distinguish these from genuine algorithmic errors, we employ a hybrid filtering pipeline combining LLM- based semantic analysis, rule-based pattern matching for known debugging idioms, and targeted manual inspection

  7. [7]

    To address this, we re-evaluate every candidate buggy submission against the current judge; any submission that now passes all test cases is discarded

    Over the years, the UOJ judging infrastructure has undergone significant upgrades, causing some older submissions (with Time Limit Exceeded verdicts) to pass under current hardware conditions. To address this, we re-evaluate every candidate buggy submission against the current judge; any submission that now passes all test cases is discarded. B.2. Token C...

  8. [8]

    Hardcode:the modification is not a genuine bug fix, but simply hardcoding: e.g., deleting debug output, adding ad-hoc checks for specific inputs, or directly returning a fixed value

Show all 15 references
  1. [9]

    Corner Case Error:Code A only fails on very small, very large, or boundary inputs (e.g., n=0, n=1, maximum values, overflow, precision issues, division by zero). 3.Careless Error: • Wrong data type (e.g., int vs long long) • Wrong variable name (e.g., swapping n and m) • Wrong...

  2. [10]

    Randomness-related Error:Code A uses randomization but fixes a random seed or parameter, so adversarial input can be constructed against it

  3. [11]

    Both Wrong

    Both Wrong:Neither code is a fully correct solution; both rely on heuristics, pruning, or parameter tuning, and should be excluded. Notice that if the new content in code B has some magic number or constant number, and these number are not for some corner case like n=1, you sh...

  4. [12]

    Context Inference:If the generated patch omits line numbers, our pipeline scans the source code to heuristically locate 16 UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming the matching context block and infers the correct insertion point

  5. [13]

    monolithic

    Header Correction:Since models often hallucinate or miscount the number of added/deleted lines in the hunk header, we ignore the generated counts. Instead, we programmatically recalculate the correct line counts based on the actual content of the patch body. 3.Fuzzy Applicatio...

    2014

  6. [14]

    This suggests that the models’ high performance on older “Hard” problems is partially attributable to memorization of canonical solutions present in the training data

    Code Generation:We observe a discernible decline in success rates for more recent problems. This suggests that the models’ high performance on older “Hard” problems is partially attributable to memorization of canonical solutions present in the training data

  7. [15]

    low-signal

    Code Hacking and Self-Repair:In contrast, these tasks exhibit a stable performance trend across the timeline. This stability indicates that Hacking and Repair rely less on retrieving memorized patterns and more on active reasoning, logic verification, and understanding specifi...

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.