{"id":"31ea2d6e-2479-40a5-9e60-3c758e02e8b5","arxiv_id":"2506.10481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A private, contamination-resistant benchmark of 250 olympiad-level programming problems shows top reasoning models reaching about 36% solve rates, far above conventional models.","lead":"OIBench is a new benchmark of 250 original, unpublished olympiad-level programming problems, released to the public for testing AI models. It adds time and space efficiency curves and human comparisons, with the goal of resisting memorization and separating strong reasoning models from weaker ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The anti-contamination evidence is weaker than claimed: the §3.1 Risk-Score experiment may not evaluate models on the 100 OIBench problems used for training, so it cannot rule out memorization-based contamination.","rationale":"OIBench is a useful benchmark, and the leaderboard separation between reasoning and non-reasoning models is informative. The main risk to the paper's central value proposition is the contamination-resistance claim. The only empirical evidence for it is the Risk-Score experiment in §3.1/A.3. The definition of Risk-Score makes the reported <0.01 values difficult to reconcile with the described protocol: SFT on 100 OIBench problems should yield measurable memorization if those problems are in the evaluation. The low values are only coherent if the contaminated model is scored on a different set (e.g., the non-training problems) or if the training format prevents retrieval. Neither interpretation supports the stated conclusion that leakage of OIBench into training data would not inflate scores. The same concern was anticipated by the reader's weakest assumption about originality and absence from pre-training corpora; this attack sharpens it by focusing on the internal validity of the leakage experiment. A secondary issue remains the contradiction between the abstract and conclusion claiming models outperform 'most' human participants and Figure 4 showing O4-mini-high above only 42.64% on average; that wording should be corrected regardless, but it is more easily fixed than the contamination claim.","tokens_in":18414,"tokens_out":9394,"duration_ms":116549,"concrete_test":"Use the released training/evaluation code to rerun the §3.1 experiment, and add two measurements: (i) evaluate the 'leaked' model on the exact 100 OIBench problems used in SFT with the same prompts and judge; (ii) evaluate it on the remaining OIBench problems. If the exact-100 pass rate is not substantially above the baseline, the near-zero Risk-Score is explained by evaluation split or format mismatch. If the exact-100 pass rate is substantially above baseline, then a low overall Risk-Score does not demonstrate contamination resistance and the claim must be scaled back.","verdict_should_be":"UNCHANGED","load_bearing_attack":"§3.1 and A.3 describe a contamination simulation in which 100 OIBench problems with official solutions are mixed into SFT data. Risk-Score is defined as (S_contaminated − S_baseline)/(1 − S_baseline), and Table 2 reports values <0.01. If those 100 problems are part of the 250-problem evaluation and the model memorizes even a fraction of them, the overall score should jump far above the low baselines (e.g., for Qwen2.5-72B, baseline ≈1.8%; memorizing 40 of 100 problems would add ≈16 points, a Risk-Score near 0.16, not 0.01). The reported near-zero values therefore indicate one of two things: (a) the evaluation set excludes the 100 SFT problems, so the experiment tests generalization from OIBench-like data rather than contamination of the benchmark itself; or (b) the training setup prevents the model from reproducing the memorized solutions. The paper does not state which. As written, this experiment does not establish that a future model trained on the publicly released OIBench problems and solutions cannot benefit from direct memorization, so the central 'contamination-resistant' claim is under-supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OIBench, a benchmark of 250 olympiad-level informatics problems collected from ACM-ICPC/OI coaches, with bilingual statements, test cases, canonical C++ solutions, difficulty labels, and a public release that includes model responses and compute costs. The evaluation covers 18 LLMs on four languages in zero-shot settings, reports AC rates on OIBench and a pseudocode-aided variant, introduces Time/Space Completion Curves for efficiency, runs an SFT contamination simulation with a Risk-Score metric, and compares model rankings with human ACM contestants on 44 problems. The main results are that reasoning models (O4-mini-high 36.35% overall) far outperform instruction-tuned models (average 3.6%), closed models outscore open models, all models remain below canonical-solution efficiency, and O4-mini-high ranks highest among evaluated models in the human comparison. The paper also proposes four derived code-understanding tasks (Bugfix, Complete, Translate, Interpret).","tokens_in":18619,"tokens_out":10078,"duration_ms":117963,"significance":"OIBench is a valuable and carefully constructed benchmark artifact. The authors verify canonical solutions by executing them on all test cases, perform search-engine originality checks, document prompts and environments, release model responses and reproduction costs, and propose a more informative efficiency visualization than a single time/space ratio. If the contamination-resistance claim survives, OIBench would fill a real gap between saturated code benchmarks and much harder competition data. The main claims in need of repair are the abstract/conclusion statement about beating 'most' human participants, which Figure 4 contradicts, and the §3.1 Risk-Score experiment, whose design as written cannot distinguish memorization from generalization.","major_comments":[{"comment":"The abstract and conclusion claim that 'current SOTA models already outperform most human participants' and that 'frontier reasoning models surpass the majority of ACM-level competitors.' Figure 4 reports average relative rankings of 42.64% for O4-mini-high, 36.86% for DeepSeek-R1, and 31.47% for O3-mini-high, with all other models below 30%. None of these values exceed 50%, so the data do not support the 'most' or 'majority' wording. Please either soften the claim to 'a substantial fraction' or report a per-problem analysis that would justify a majority interpretation.","section":"Abstract / §6 / Fig. 4"},{"comment":"The Risk-Score experiment does not establish contamination resistance as written. A.3 states that the 100 problems mixed into the SFT data are '100 samples from OIBench,' but §3.1 does not state whether these 100 problems are part of the 250-problem evaluation set. If they are included, a model that memorizes even a fraction of them should raise the overall AC rate far above the 1–4% baselines; for Qwen2.5-72B, memorizing 40 of the 100 problems would add roughly 40/250 = 16 percentage points, implying a Risk-Score near 0.16 rather than the reported 0.01. If they are excluded, the experiment tests generalization from OIBench-like data rather than contamination of the benchmark itself. The paper must clarify the evaluation-set composition, report S_baseline and S_contaminated on the exact evaluation set, and either include the 100 SFT problems in the evaluation or explicitly exclude them and state the consequence. Without this, the central 'contamination-resistant' claim in the abstract and §1 is under-supported, as the authors themselves partly acknowledge in A.1 regarding RL-stage contamination.","section":"§3.1 / A.3 / Table 2"}],"minor_comments":[{"comment":"The sentence 'By default, we give all the information' is incomplete; finish it or delete it.","section":"§4"},{"comment":"The difficulty comparison mixes pass@1 results (HumanEval, OIBench) with pass@8 results (CodeElo) and results from different sources; state the metrics explicitly and consider reporting a single metric for comparability.","section":"Table 4"},{"comment":"The paper says R1 converts canonical solutions into pseudocode, but it does not describe how the pseudocode was validated to be language-agnostic and faithful to the intended algorithm; please add a verification step or at least a manual-check description.","section":"§4.2"},{"comment":"The violin plots do not specify how many human participants attempted each of the 44 problems or how ties in the IOI-style ranking were broken; adding per-problem participant counts and tie-breaking rules would improve reproducibility.","section":"§5.1 / Fig. 4"},{"comment":"The column 'avg. size of test per prob' does not state the unit; specify whether it is bytes, number of test cases, or something else, since OIBench's value of 37.5M is orders of magnitude larger than the others.","section":"Table 1"},{"comment":"There are numerous typos, including 'Learderboards' in the §4 heading, 'fine-grand' in §4.3, 'inprovment' in §4.3, 'serveral' and 'erros' in §5.2, 'elaborted' in the baseline-model description, 'Univerisity' in the affiliation, and 'Comparsion'/'resutles' in A.2; a light copyedit is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The benchmark artifact and execution pipeline are strong, and the release of problems, test cases, model responses, and costs is a significant community contribution. The main obstacles are the mismatch between the abstract/conclusion and Figure 4, and the uninterpretable Risk-Score experiment; both are fixable in revision, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OIBench is a useful addition to the code-benchmark ecosystem, and you should treat the resource itself as real. Two claims in the paper, though, are stronger than the evidence: the 'contamination-resistant' tag and the claim that SOTA models 'outperform most human participants.'\n\nWhat's actually new: 250 private olympiad-level problems from experienced coaches, with search-engine verification, robust test cases, canonical solutions, and a full open release (problems, solutions, model responses, costs). That's a genuinely valuable evaluation set. The Time/Space Completion Curves are a simple but nice way to visualize efficiency; they're essentially CDFs of runtime/memory, so the methodological novelty is modest. The static human-comparison protocol on 44 problems is a real improvement over CodeElo's online Elo approach, and releasing the anonymized human records is a good reproducibility practice. The leaderboard separation between reasoning and instruction-tuned models (O4-mini-high 36.35% vs ~3.6% average) makes the benchmark discriminating.\n\nSoft spots. First, the anti-contamination experiment in §3.1 doesn't close the loop. The paper mixes 100 OIBench problems into SFT data but never says whether those 100 are part of the 250-problem evaluation. If they're excluded, the experiment only shows that SFT on OIBench-like problems doesn't transfer—it doesn't test whether a model trained on the released problems would memorize them. If they're included, near-zero Risk-Scores are implausible; even a small number of memorized solutions would move the overall score well above the 1-4% baselines. The authors need to state the split and, ideally, report the leaked models' scores on the 100 seen problems directly. The limitation section does admit RL-stage contamination is untested, which is honest, but the abstract's 'contamination-resistant' claim is unqualified.\n\nSecond, the abstract and conclusion claim that current SOTA models 'outperform most human participants.' Figure 4 shows the best model (O4-mini-high) beating 42.64% of humans on average—not a majority. The conclusion's 'surpass the majority of ACM-level competitors' has the same problem. This is a fixable wording issue, but it matters because the human comparison is a headline contribution.\n\nWho should read this: anyone building or using code-reasoning benchmarks. The dataset itself is citable now; the paper needs a revision before its claims are. I'd send it to peer review, with a request to clarify the contamination experiment and correct the human-comparison language.","headline":"OIBench is a genuinely useful benchmark resource, but the paper's two headline claims—contamination resistance and beating 'most' human contestants—are both stronger than the evidence supports.","tokens_in":19169,"tokens_out":4367,"would_cite":true,"duration_ms":49129,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OIBench is a contamination-resistant benchmark of 250 original olympiad-level programming problems that separates reasoning models (21.4% average) from conventional instruction-tuned models (3.6%), while still showing an efficiency gap to…","keywords":["OIBench","olympiad in informatics","code generation benchmark","benchmark contamination","reasoning models","test-time scaling","algorithmic efficiency","human-model comparison"],"falsifier":"Check the released 250 problems against a large pre-training corpus and public code dumps before release; finding any near-verbatim match would falsify the originality claim. Independently, run the paper's contamination experiment with reinforcement learning instead of supervised fine-tuning, training a strong base model on 100 held-out problems and measuring the solve-rate jump on the remaining 150; a jump far above the reported Risk-Score below 0.01 would falsify the contamination-resistance claim.","tokens_in":18220,"feed_emoji":"🏆","tokens_out":9599,"duration_ms":109032,"temperature":0.7,"pith_summary":"OIBench is an attempt to build a coding benchmark that cannot be gamed by memorization: 250 original, bilingual, olympiad-level informatics problems written by experienced competition coaches and checked to be absent from the public web before release. The paper's central claim is that, on this private test, models that do extended reasoning before answering solve about 21.4% of problems on average, versus about 3.6% for conventional instruction-tuned models, with the strongest evaluated model reaching 36.35%. It also introduces Time/Space Completion Curves to show how close generated code comes to the canonical solution's time and memory use, and finds that even the best models remain substantially less efficient. On a 44-problem subset, frontier models outperformed the average human contestant while still trailing top human players. If the problems are as original as claimed, the benchmark offers a discrimination signal that saturated pass-rate benchmarks no longer provide.","feed_headline":"Secret olympiad problems: a 21% vs 3.6% reasoning gap","feed_subtitle":"Private benchmark: even top models trail canonical solutions and beat only average human contestants.","key_machinery":"The central instrument is the Time/Space Completion Curve, a cumulative distribution that plots the fraction of test cases a model's submissions pass against the maximum time or memory used, expressed as a percentage of the canonical solution's usage on a log scale. This turns a binary pass/fail acceptance rate into a detailed view of algorithmic efficiency. The benchmark itself is the second mechanism: 250 original problems from experienced competition coaches, each with large test cases, a verified C++ canonical solution, bilingual statements, and difficulty labels, checked for absence from the web. The Risk-Score contamination metric, defined as the improvement from adding in-distribution problems to supervised fine-tuning divided by the remaining headroom, is the third mechanism, quantifying how much a leaked training set could inflate scores.","core_discovery":"The paper establishes that the frontier in code generation is not on memorized, easy tasks but on unseen, hard problems where a model must discover the right algorithm and implement it efficiently. Its leaderboard shows a clear separation between reasoning models and conventional models, while the pseudocode variant reveals that giving a solution outline narrows the gap, placing much of the difficulty in problem discovery rather than code writing. The Time/Space Completion Curves show that all models' generated code uses more time and memory than the canonical C++ solutions, even at the same correctness level. The human comparison places current state-of-the-art models above most ACM-level contestants but below the strongest humans, and the paper's contamination experiments report a Risk-Score below 0.01, indicating that supervised fine-tuning on in-distribution problems does not inflate scores.","pith_inferences":["Beyond the paper: the completion-curve method transfers to any generation task with resource budgets, such as tokens used per correct math answer or memory used per data-science pipeline, as a finer-grained alternative to accuracy-only reporting.","Beyond the paper: since the contamination experiment covers supervised fine-tuning but not reinforcement learning, the benchmark's long-term value depends on a refresh protocol; running the Risk-Score experiment with reinforcement learning on held-out problems would be the decisive follow-up.","Beyond the paper: the bimodal \"peanut-shaped\" human-rank distribution suggests current long-chain-of-thought models specialize on some problem types and fail on others; grouping the human-comparison problems by algorithm tag would test whether failures cluster.","Beyond the paper: the observation that conventional models do relatively better at very small time budgets hints that they fall back on memorized naive algorithms that are fast only on small test cases, which is testable by comparing their solutions' complexity classes across problem sizes."],"forward_implications":["Passing a coding benchmark's acceptance tests no longer guarantees competitive efficiency; reporting completion curves alongside pass rates should become standard.","Because pseudocode hints lift scores even for the strongest models, future progress in code reasoning will come as much from discovering the right algorithm as from writing code.","The leading model's superiority comes with better reasoning efficiency, solving more problems per inference token, making long-chain-of-thought efficiency a concrete optimization target.","Static, controlled human comparisons on a fixed problem set can serve as a reproducible alternative to months-long online Elo evaluation.","Open-source models trail closed-source ones, but the strong result of a reasoning-distilled model suggests that distilling chain-of-thought into conventional models transfers some reasoning ability."],"supporting_citations":[{"why":"Defines the HumanEval benchmark whose near-saturation motivates the need for harder coding benchmarks.","marker":"[1]"},{"why":"Provides the MBPP benchmark, another saturated conventional coding test used as a baseline for difficulty.","marker":"[2]"},{"why":"Introduces CodeElo's online Elo comparison approach that OIBench replaces with a static human-comparable evaluation.","marker":"[6]"},{"why":"Supplies the USACO olympiad benchmark that OIBench extends with private, harder, original problems.","marker":"[7]"},{"why":"Establishes LiveCodeBench's low-contamination evaluation design that OIBench's privacy checks build on.","marker":"[8]"},{"why":"Proposes EffiBench's runtime-and-memory efficiency metrics that the Time/Space Completion Curves refine.","marker":"[9]"},{"why":"Reports DeepSeek-R1, a central reasoning-model baseline in the leaderboard and error analysis.","marker":"[40]"},{"why":"Provides OpenAI O1 as another central reasoning-model baseline and test-time-scaling comparison.","marker":"[12]"},{"why":"Supplies the statistical method for confidence intervals used to support the leaderboard comparisons.","marker":"[41]"}],"fun_headline_variants":["SOTA models beat most humans but trail canonical solutions","New benchmark: AI solves olympiad problems, still slower than optimal","Secret olympiad test: models outthink humans, not yet optimal","Code reasoning benchmark: SOTA beats humans, lags optimal solutions","OIBench: AI outperforms contestants, misses canonical code efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claims rest on the assumption that all 250 problems were genuinely unseen by the evaluated models during training; the support is coach attestation, web-search checks, and a fine-tuning experiment that does not cover reinforcement-learning-phase contamination.","fun_headline_variants_meta":{"raw":{"variants":["SOTA models beat most humans but trail canonical solutions","New benchmark: AI solves olympiad problems, still slower than optimal","Secret olympiad test: models outthink humans, not yet optimal","Code reasoning benchmark: SOTA beats humans, lags optimal solutions","OIBench: AI outperforms contestants, misses canonical code efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2107,"prompt_tokens":877,"completion_tokens":1230,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1142}},"tokens_in":493,"tokens_out":1230,"duration_ms":12431,"temperature":1.0,"reasoning_tokens":1142,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:25:02.808209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the released 250 problems against a large pre-training corpus and public code dumps before release; finding any near-verbatim match would falsify the originality claim. Independently, run the paper's contamination experiment with reinforcement learning instead of supervised fine-tuning, training a strong base model on 100 held-out problems and measuring the solve-rate jump on the remaining 150; a jump far above the reported Risk-Score below 0.01 would falsify the contamination-resistance claim.","supporting_citations":[{"cited_title":"Effibench: Bench- marking the efficiency of automatically generated code","cited_arxiv_id":null,"evidence_quote":"Proposes EffiBench's runtime-and-memory efficiency metrics that the Time/Space Completion Curves refine."}],"review_version":1}