REVIEW 3 major objections 5 minor 1 cited by
LiveOIBench — a 403-problem Informatics Olympiad benchmark with official test cases and human contestant results — finds that GPT-5, the top LLM evaluated, reaches the 81.76th human percentile but still falls short of elite human competitor
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
2026-08-04 10:24 UTC pith:JA3OGKSN
load-bearing objection A solid, useful benchmark with careful evaluation; the headline human-percentile numbers need re-framing from best-of-8 oracle selection to a single-attempt or clearly labeled protocol. the 3 major comments →
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On a new, high-quality benchmark of 403 expert-curated Informatics Olympiad problems from 2023–2025, no evaluated LLM surpasses elite human contestants. GPT-5 achieves the highest performance with an average human percentile of 81.76 and a pass rate of 63%, while the best open-weight model, GPT-OSS-120B, reaches only the 59.90th percentile. The benchmark integrates official test cases, subtask rubrics, and contestant score distributions, enabling direct human-model comparison via percentiles, medal thresholds, and Codeforces Elo ratings. Reasoning-trace analysis indicates that higher-performing models allocate more tokens to focused problem analysis and planning, and less to exploratory back
What carries the argument
LiveOIBench itself is the central object: a dataset of 403 problems from 72 contests across 14 Informatics Olympiads, each with an average of 60 official test cases, subtask scoring rubrics, and linked human contestant results. This design allows models to be scored with official judges and compared to humans using percentile ranks, medal cutoffs, and Codeforces ratings. A second mechanism is the reasoning-trace taxonomy—categorizing model reasoning into analysis, planning, exploration, implementation, and verification—which the paper uses to correlate cognitive behavior with solution correctness.
Load-bearing premise
The human-comparison results hinge on the correctness and completeness of the crawled official test cases and contestant-score data, and on the assumption that the 2023–2025 problems were not memorized by the evaluated models—a premise the paper supports only indirectly.
What would settle it
Release a fresh, never-seen Olympiad contest through the same pipeline within days of its administration, evaluate GPT-5 and GPT-OSS-120B on it, and compare their human percentiles to those reported; a sharp drop on the fresh contest would indicate that contamination inflated the published scores.
If this is right
- LiveOIBench provides a reproducible, offline standard for tracking future LLM progress on Olympiad-level programming, avoiding reliance on external judge APIs.
- The human-percentile comparison shows a clear, quantified gap between frontier models and top human contestants, establishing a concrete target for the field.
- The subtask rubrics allow partial-credit evaluation, revealing that models struggle most on later, less-constrained subtasks and on algorithms like dynamic programming and tree problems.
- Reasoning-trace findings suggest training and inference strategies that encourage structured analysis and limit exploratory backtracking may yield better coding performance.
Where Pith is reading between the lines
- The 81.76 percentile likely overstates true generalization if any 2023–2025 problem text leaked into proprietary training sets; the paper's contamination checks are indirect for closed models, so the human-relative numbers are best read as upper bounds.
- The reasoning-trace pattern—stronger models explore less and analyze more—could motivate test-time techniques that dynamically cap backtracking, though the paper does not itself test such interventions.
- Because the benchmark is updated with each new Olympiad season, it could serve as a living 'human-vs-machine' leaderboard, akin to Elo ratings, for the competitive programming community.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. LiveOIBench introduces a benchmark of 403 expert-curated Informatics Olympiad problems from 72 contests held in 2023–2025, with official test cases, subtask rubrics, and human contestant results for 46 of the contests. The authors evaluate 34 LLMs using best-of-8 C++ sampling, reporting that GPT-5 reaches an 81.76 human percentile and that the best open-weight model, GPT-OSS-120B, reaches 59.90. The paper also presents analyses by algorithm tag, reasoning-trace behavior, error patterns, and data-contamination checks, and releases data, code, and a leaderboard.
Significance. If the evaluation protocol is sound, LiveOIBench is a valuable contribution: it uses official private test cases (reducing false positives relative to generated tests), provides fine-grained subtask scoring, directly links model scores to human contestant rankings, and ships an offline judge with verified official solutions. The 2023–2025 time window is a sensible contamination mitigation. The main caveats are that the headline human-comparison numbers are computed under an oracle-selected best-of-8 protocol, that the denominator for human percentiles is ambiguous (46 vs 72 contests), and that the contamination analyses are indirect. These issues are fixable with re-analysis or reframing, but they affect the paper's central claims.
major comments (3)
- [§4, Table 2] The headline human-percentile numbers are computed under a best-of-8 protocol: 'we sample 8 candidate solutions per model in C++ and pick the solution with the highest score' (Section 4). Because the highest score is selected by running all candidates against the official test set, the reported percentiles (e.g., GPT-5 81.76, GPT-OSS-120B 59.90) measure an oracle-selected upper envelope, not single-attempt skill. A human contestant has no such oracle and is constrained to one contest trajectory. The claim that GPT-5 'falls short of top human contestants' is only meaningful for this best-of-8 setting. Please report Pass@1 / single-sample human percentiles, or explicitly label all human-comparative metrics as best-of-8 with the selection rule.
- [Appendix A.1 vs Table 2 / Table A9] Appendix A.1 states that 46 of the 72 contests include human contestant results, yet Table 2 and Table A9 report Human Percentile and ELO for 'all 72 contests' (Table 2 caption). This is a denominator problem: for the 26 contests without official human results, how is a human percentile defined? If those contests are excluded from human-comparative metrics, the table should say so; if they are included with some imputation, the imputation must be described. The current presentation makes the central human-comparison numbers unverifiable.
- [§5.4.2, Appendix E.8] The contamination analysis uses GPT-OSS-120B's token-level familiarity as the only model-specific signal (Section 5.4.2) and applies it to draw conclusions about all 34 models, including closed-weight GPT-5. Familiarity measured by one open-weight model does not test whether a different proprietary model memorized the 2023–2025 problems; temporal trends and code similarity are indirect and low-power. The abstract's 'minimal evidence of data contamination' is therefore stronger than the evidence supports. Please either provide per-model contamination checks (for open-weight models, direct perplexity/familiarity; for closed models, API-based or behavioral probes) or weaken the claim to 'no evidence in the checks we ran.' Relatedly, Appendix E.8 uses GPT-OSS-120B to annotate GPT-OSS-120B reasoning traces; with 86% manual agreement and 10% missed behaviors, Section 5.2's behavioral conclusio
minor comments (5)
- [Table 2 / Appendix C] Clarify that the Pass Rate column is Pass@8 and that all percentiles are best-of-8; currently 'Pass Rate' is only formally defined in Appendix C, while Table 2 leaves the reader to infer the sampling protocol.
- [Figure 3 caption] The caption says 'eight reasoning behaviors' but groups them into five; please explicitly list the eight behaviors and the grouping in the caption for readability.
- [Appendix E.8] The chunking description is inconsistent: 'segments of approximately 5k tokens, estimated by dividing the total token length by four.' Specify the actual chunking procedure and whether chunks were fixed-size or variable.
- [Appendix A.4] The Elo formula uses 'r' and 'r(i)' with inconsistent notation. Also, the outlier filtering threshold (2 standard deviations) and the 15-valid-rating exclusion rule should be justified or accompanied by a sensitivity analysis.
- [Table A5] USACO contests are listed with Human Results=False, yet USACO problems are included in the main results. State explicitly how USACO contests are handled in human-percentile and ELO calculations, since official human results are not available for them.
Circularity Check
No significant circularity: the benchmark's central results are external measurements; the only self-referential element is the GPT-OSS-120B trace annotation, which is validated and not load-bearing.
full rationale
We walked the paper's derivation chain and found no step where a claimed result reduces to its inputs by construction. LiveOIBench's construction is grounded in externally crawled official problems, test cases, subtask rubrics, and human contestant results; model scores are obtained by running generated C++ solutions against those test cases in a local judge. The human percentile, medal, pass rate, and Codeforces Elo metrics are direct score-based mappings to external human data, not fitted parameters that are then relabeled as predictions. Contamination analyses use external evidence: release-date trends, Dolos code-similarity comparisons, and familiarity correlations; the familiarity check is computed with GPT-OSS-120B on GPT-OSS-120B, but it is a measurement of that model's own token probabilities, not a parameter fitted to the outcome being claimed. The only self-referential element is the reasoning-trace analysis (Section 5.2 / Appendix E.8), where GPT-OSS-120B annotates GPT-OSS-120B traces; however, the paper provides manual validation (86% accuracy on 313 behaviors) and Claude-based inter-model agreement, and this analysis is auxiliary rather than central to the benchmark's headline claims. The best-of-8 oracle selection (Section 4) is a human-comparability caveat, not a circularity, since the model's score is still independently computed. No load-bearing self-citations or imported uniqueness theorems are present. Overall, the central benchmark and comparison results are self-contained external evaluations.
Axiom & Free-Parameter Ledger
free parameters (5)
- sample count k =
8
- decoding temperature/top-p per model =
varies by model, e.g., 0.6/0.95 for Qwen3, 1.0/1.0 for GPT-OSS
- reasoning budget (max tokens) =
e.g., GPT-5 Medium 100K; GPT-OSS 128K; Qwen3 38K
- outlier filtering threshold in ELO calibration =
2 standard deviations from a cubic fit
- difficulty bucket cut points =
Easy d<=17; Medium 18-22; Hard d>=23
axioms (6)
- domain assumption Official contest test cases, subtask rubrics, and judge behavior are correct and complete as distributed by organizers or CSES/LibreOJ.
- domain assumption Human contestant score files and Codeforces profile matches are accurate and complete enough for percentile and ELO calculations.
- domain assumption The CodeElo formula from Quan et al. (2025) is a valid way to convert contest performance to Codeforces ratings.
- domain assumption Familiarity measured by GPT-OSS-120B's token probabilities is a meaningful proxy for contamination in all evaluated models.
- domain assumption PDF-to-markdown conversion followed by Gemini-based correction preserves problem semantics.
- domain assumption Sampling 8 candidates and taking the best is an appropriate way to compare LLMs with human contestants.
Cite this review
Pith. "Pith review of LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?." pith.science (2026). https://pith.science/paper/JA3OGKSN
@misc{pith2026251009595,
author = {Pith},
title = {Pith review of: LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?},
year = {2026},
howpublished = {\url{https://pith.science/paper/JA3OGKSN}},
note = {Machine review of arXiv:2510.09595}
}
read the original abstract
Competitive programming problems are increasingly used to evaluate the coding capabilities of large language models (LLMs) due to their complexity and ease of verification. Yet, current coding benchmarks face limitations such as a lack of exceptionally challenging problems, insufficient test case coverage, and reliance on online platform APIs that limit accessibility. To address these issues, we introduce LiveOIBench, a large-scale competitive programming benchmark featuring 403 expert-curated problems, averaging 60 official test cases each, drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025. LiveOIBench has four key features: (1) expert-designed tasks with detailed subtask rubrics and extensive test cases; (2) direct comparison to elite human contestants; (3) continuous updates to reduce contamination risk; and (4) a fully offline, reproducible evaluation system. Benchmarking 34 popular general-purpose and reasoning LLMs, we find that GPT-5 achieves an 81.76th percentile, still falling short of top human contestants, while among the open-weight models, GPT-OSS-120B reaches only the 60th percentile. Reasoning-trace analyses indicate that robust reasoning models prioritize precise problem analysis over excessive exploration. Finally, analyses across release dates, task familiarity, and code similarity find minimal evidence of data contamination in our benchmark. Our leaderboard, code, and data are available at: https://liveoibench.github.io/.
Figures
Forward citations
Cited by 1 Pith paper
-
The FIL Hypothesis: Inductive Biases Help with Kernel Engineering
The FIL Hypothesis claims that inductive biases outperform purely data-driven methods on GPU programming tasks with non-trivial feedback loops.
Reference graph
Works this paper leans on
-
[2]
(7 lines omitted) [3][Planning]Goal: Find multiset of popcounts (c_i) (0 <= c_i <= 30) such that xor of c_i = K and sum of some numbers with those popcounts equals M
-
[4]
(1 line omitted) [5][Analysis]However we can realize any popcount c can be realized by a number with value at most(2 c −1)(e.g., lower bits)
-
[6]
(86 lines omitted) [7][Planning]Thus problem reduces to: For given desired popcount c and target value S (0<=S<=M), does there exist a non-negative integer x with popcount(x)=c and x=S?
-
[8]
44 Preprint
(46 lines omitted) [9][Analysis]Thus we can start with binary decomposition of M into bits: we have list L of powers-of-two (size = popcount(M)). 44 Preprint
-
[10]
(19 lines omitted) [11][Analysis]If we split a number with popcount c into c ones: before popcount = c, after xor = (1 xor 1
-
[12]
(29 lines omitted) [13][Implementation]Implementation plan: - Decompose M into list of powers-of-two bits (each value =2 i).[INCORRECT]
-
[14]
M=10 (1010) popcount=2, X=0
(28 lines omitted) [15][Verification]Let’s test: starting X parity = (number of bits) %2 = popcount(M) %2. M=10 (1010) popcount=2, X=0. Target K=5 (101) delta=5 xor 0 =5 (odd). Since delta odd cannot be achieved by merging distinct bits (even changes). So maybe impossible, matching sample
-
[16]
[18][Verification]M=33 (100001) popcount=2 even, K=5 odd => different parity, but sample says possible!
(9 lines omitted) [17][Verification]Check other samples: M=2 (10) popcount=1 odd, K=1 odd -> same parity -> possible. [18][Verification]M=33 (100001) popcount=2 even, K=5 odd => different parity, but sample says possible!
-
[19]
Ensure c >=0
(33 lines omitted) [20][Analysis]If not found with 2 numbers, try 3 numbers: enumerate a,b from set S (size few thousand) and compute c = M-a-b, check if pc(c) = K xor pc(a) xor pc(b). Ensure c >=0. This isO(|S| 2)~ few million per test, okay for 5k?
-
[21]
Define processing bits from LSB to MSB with borrow flag (0/1)
(22 lines omitted) [22][Implementation]Idea: Use DP over bits to construct a with given popcount p such that popcount(M-a)=q.[INCORRECT] [23][Analysis]We can DP over bits with carry (borrow) state. Define processing bits from LSB to MSB with borrow flag (0/1). For each bit i, we decidea i (0/1). Thenb i = (Mi −a i −borrow)mod 2, and new borrow = (a i + bo...
-
[24]
Initialize dp[0][0][0][0] = true
(11 lines omitted) [25][Analysis]Implement DP bool dp[pos][borrow][pa][pb] reachable. Initialize dp[0][0][0][0] = true. Iterate bits i from 0..30 (sinceM <2 30). At each step, knownM i (0/1). Fora i in {0,1}. Then compute diff =M i −a i −borrow. bi = (diff & 1) (i.e., (diff+2)%2). 45 Preprint newBorrow = diff<0 ? 1 : 0. Update counts. At end pos=30, requi...
-
[26]
[28][Analysis]If there exists pa such that pa xor pb == K, then we have solution with two numbers
(2 lines omitted) [27][Analysis]Then we have set of reachable (pa,pb). [28][Analysis]If there exists pa such that pa xor pb == K, then we have solution with two numbers
-
[29]
[31][Exploration]If no such pair, we can try three numbers? [32][Exploration]If no such pair, we can try three numbers?
(1 line omitted) [30][Analysis]We can also reconstruct by storing predecessor. [31][Exploration]If no such pair, we can try three numbers? [32][Exploration]If no such pair, we can try three numbers?
-
[33]
(6 lines omitted) [34][Exploration]But we can extend DP to three numbers?
-
[35]
Given time, maybe we can produce algorithm: Use DP to find two numbers (a,b)
(8 lines omitted) [36][Exploration]Simplify: Use DP to find any multiset of up to 5 numbers (since N<=100) with small popcounts by using DP over count of numbers? Too heavy. Given time, maybe we can produce algorithm: Use DP to find two numbers (a,b)
-
[37]
(1 line omitted) [38][Analysis]Given time, maybe we can produce algorithm: Use DP to find two numbers (a,b)
-
[39]
(12 lines omitted) [40][Exploration]if not found, output -1.[INCORRECT]
-
[41]
implementation steps
(29 lines omitted) [42][Analysis]Complexities: dp size 31*2*32*32 63k bool, fine. [43][Implementation]Implement predecessor arrays as struct of 5 bits maybe small. We’ll store prev[bit][borrow][pa][pb] with struct containing prevBorrow, prevPa, prevPb, a_bit. Also b_bit can be derived. During transition we need b_bit which depends on M_i, a_bit, borrow. W...
-
[2024]
Input" formatted as # Input while
URLhttps://arxiv.org/abs/2412.16720. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat...
Pith/arXiv arXiv 2025
-
[9203]
URL http://dx.doi.org/10.1126/science
doi: 10.1126/science.abq1158. URL http://dx.doi.org/10.1126/science. abq1158. Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, 14 Preprint Sv...
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.