Pith. sign in

REVIEW 2 major objections 1 cited by

Run-level pass rates for LLMs on programming tasks overstate retry-free coverage by up to 17.8 percentage points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 18:46 UTC pith:R7IFGXYQ

load-bearing objection The paper quantifies how single-run pass rates overstate consistent success on code tasks by up to 17.8 points and can flip rankings, but five runs per problem leave those numbers noisy. the 2 major comments →

arxiv 2606.00920 v1 pith:R7IFGXYQ submitted 2026-05-30 cs.LG cs.AIcs.SE

Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks

Yongxi Zhou , Lai Yun Choi , Jiaxi Wen , Wenbo Ye This is my paper
classification cs.LG cs.AIcs.SE
keywords LLM evaluationcode generationrun-level accuracyretry-free coveragemodel stabilityrepeated runsprompt effectsdeterministic tasks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper measures how consistently large language models solve the same deterministic programming problems when invoked repeatedly under identical prompts. Standard accuracy reporting counts every successful run separately, but this exceeds the share of problems solved correctly on every run without any retries. The difference reaches 17.8 points and is largest for mid-tier models, enough to reverse rankings between otherwise similar systems. The evaluation uses 100 LeetCode-style problems, 16 models, two prompt templates, and five runs each to separate average success from per-problem stability. Readers care because many deployed uses require the model to produce the correct output reliably rather than sometimes.

Core claim

On a recency-based benchmark of 100 LeetCode-style problems, 16 models from five provider families were evaluated under two prompt templates with five repeated runs per problem. Run-level pass rate consistently exceeds retry-free coverage, with the gap reaching 17.8 percentage points and largest for mid-performing systems; this discrepancy reverses model rankings even among closely matched systems. Run-level pass rate and perfect stability rate correlate strongly (r=0.985), yet the systematic overstatement remains. Prompt effects are model-dependent rather than uniformly beneficial. These findings indicate that repeated-run stability analysis is a necessary complement to conventional accurac

What carries the argument

The repeated-run evaluation protocol that separately tracks run-level accuracy, retry-free coverage (problems solved correctly on every run), and per-problem variability across five executions per problem.

Load-bearing premise

The 100 LeetCode-style problems together with the two prompt templates and five repeated runs per problem are sufficient to reveal the general accuracy-stability relationship for deterministic text-conditioned generation tasks.

What would settle it

Finding no systematic gap between run-level pass rate and retry-free coverage on a larger or differently constructed set of deterministic problems would falsify the claimed overstatement.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Run-level pass rate and perfect stability rate remain strongly correlated yet differ by as much as 17.8 points.
  • The overstatement gap is largest precisely for mid-performing models.
  • Prompt template choice produces model-dependent rather than uniform effects on both accuracy and stability.
  • Stability analysis must be reported alongside accuracy for deterministic generation tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same run-versus-coverage gap may appear in other deterministic text-generation domains such as structured data extraction or formal verification.
  • Benchmark designers could adopt dual reporting of both metrics to prevent misleading single-number rankings.
  • Applications that penalize inconsistent outputs may favor models with smaller gaps even when their average accuracy is lower.
  • Increasing the number of runs per problem beyond five could sharpen estimates of per-problem variability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The manuscript reports an empirical evaluation of 16 LLMs on 100 LeetCode-style deterministic programming problems using two prompt templates and five independent runs per problem (16,000 total instances). It claims that run-level pass rate consistently exceeds retry-free coverage (problems solved on all five runs) by up to 17.8 percentage points, with the gap largest for mid-performing models; the two metrics are strongly correlated (r=0.985) yet produce ranking reversals; and prompt effects are model-dependent rather than uniformly beneficial.

Significance. If the central empirical findings hold after statistical scrutiny, the work provides concrete evidence that conventional single-run or eventual-success accuracy metrics can overstate reliability for deterministic text-conditioned tasks, particularly for mid-tier models. The scale of the repeated-run protocol and the direct counts of pass rates versus perfect-stability coverage constitute a useful, falsifiable dataset for the community.

major comments (2)
  1. [Abstract] Abstract and (presumed) Results section: the reported maximum gap of 17.8 percentage points and the occurrence of ranking reversals are given as point estimates with no bootstrap confidence intervals, standard errors, or paired statistical tests on the per-problem differences. With exactly five runs per problem, the binomial variance on the all-success probability is large and the metric is highly sensitive to small changes in underlying success rate; without uncertainty quantification it is impossible to determine whether the observed gaps or reversals exceed sampling variability.
  2. [Evaluation protocol] Evaluation protocol (described in abstract): the decision to use precisely five runs is presented without power analysis, sensitivity checks, or justification relative to the binomial variance of the retry-free coverage metric. This choice directly affects the magnitude and reliability of the 17.8 pp gap that forms the central claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and for highlighting the need for greater statistical rigor in reporting our empirical results. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract and (presumed) Results section: the reported maximum gap of 17.8 percentage points and the occurrence of ranking reversals are given as point estimates with no bootstrap confidence intervals, standard errors, or paired statistical tests on the per-problem differences. With exactly five runs per problem, the binomial variance on the all-success probability is large and the metric is highly sensitive to small changes in underlying success rate; without uncertainty quantification it is impossible to determine whether the observed gaps or reversals exceed sampling variability.

    Authors: We agree that uncertainty quantification is necessary to assess whether the reported gaps and ranking reversals exceed sampling variability. In the revised manuscript we will add bootstrap confidence intervals (with 10,000 resamples) for the per-model differences between run-level pass rate and retry-free coverage, and we will apply paired McNemar tests on the per-problem binary outcomes to evaluate statistical significance of the observed differences. revision: yes

  2. Referee: [Evaluation protocol] Evaluation protocol (described in abstract): the decision to use precisely five runs is presented without power analysis, sensitivity checks, or justification relative to the binomial variance of the retry-free coverage metric. This choice directly affects the magnitude and reliability of the 17.8 pp gap that forms the central claim.

    Authors: The choice of five runs was driven by the need to keep the total evaluation budget (16,000 instances) tractable while still permitting estimation of stability. No formal power analysis was performed prior to data collection. In revision we will add a post-hoc sensitivity analysis that recomputes the gap, correlation, and ranking reversals using the first 3 and first 7 runs of the existing data, together with a brief justification tied to the observed per-problem variance in our pilot runs. revision: partial

Circularity Check

0 steps flagged

No circularity; purely empirical counts from repeated runs

full rationale

The paper reports direct empirical measurements: run-level pass rate, retry-free coverage, and their differences are computed as simple averages and fractions over the 16,000 binary outcomes from five runs on each of 100 problems. No equations, fitted parameters, self-citations, or derivations are present; the reported 17.8pp gap and correlation (r=0.985) are observed statistics, not constructed by definition or reduced to inputs. The evaluation protocol is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard assumptions about problem determinism and sampling independence; no free parameters or new entities are introduced.

axioms (1)
  • domain assumption Programming problems have deterministic, unique correct solutions that can be checked automatically.
    Required for defining pass rate and retry-free coverage on LeetCode-style tasks.

pith-pipeline@v0.9.1-grok · 5759 in / 1103 out tokens · 27167 ms · 2026-06-28T18:46:14.458688+00:00 · methodology

0 comments
read the original abstract

Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points -- and the gap is largest precisely for mid-performing systems. We investigate this accuracy--stability relationship in large language model (LLM) evaluation for deterministic text-conditioned generation, using programming tasks as a concrete testbed. Standard code-generation benchmarks emphasize single-run accuracy or eventual success under repeated sampling, but many deployment settings also require stability: consistent outcomes across repeated invocations under the same task description. We present a repeated-run evaluation protocol with metrics for run-level accuracy, retry-free coverage, and per-problem variability. On a recency-based benchmark of 100 LeetCode-style problems, we evaluate 16 models from five provider families under two prompt templates with five repeated runs per problem, yielding 16,000 evaluation instances. Although run-level pass rate and perfect stability rate are strongly correlated (r=0.985), pass rate consistently exceeds retry-free coverage -- a gap that reaches 17.8 percentage points and reverses model rankings even among closely matched systems. Prompt effects are model-dependent rather than uniformly beneficial. These results suggest that repeated-run stability analysis is a necessary complement to conventional accuracy reporting for deterministic text-conditioned generation tasks.

Figures

Figures reproduced from arXiv: 2606.00920 by Jiaxi Wen, Lai Yun Choi, Wenbo Ye, Yongxi Zhou.

Figure 1
Figure 1. Figure 1: RLPR vs. PSR scatter plot across all 32 con [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cross-model stability heatmap (DETAILED prompt). Rows: 16 models sorted by RLPR descending. Columns: 100 problems grouped by difficulty tier with dashed separators, and ordered within each tier by decreasing mean pass rate. Cell color encodes empirical pass rate over R = 5 runs (green = 1.0, red = 0.0, yellow ≈ 0.5) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Problem-level success distribution (DETAILED prompt). The figure contains one panel per model (16 panels total), arranged in decreasing order of PSR. Each panel shows the fraction of that model’s 100 problems falling in each empirical pass-probability bin pˆi ∈ {0/5, 1/5, . . . , 5/5}, where 0/5 means the model failed all 5 runs and 5/5 means it passed all 5. gap than expected at the model’s accuracy level… view at source ↗
Figure 5
Figure 5. Figure 5: Prompt sensitivity: paired scatter of PSR [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 3
Figure 3. Figure 3: Computational experiments 12. If you ran experiments, did you report the number of parameters for each model? No. Parameter counts for the evaluated mod￾els are not publicly disclosed by all provider families (OpenAI, Anthropic, Google, Qwen, DeepSeek). The paper reports the API identi￾fiers, model tier designations, and access dates in [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?

    cs.SE 2026-07 conditional novelty 6.0

    On 116 LiveCodeBench tasks, Claude reviewing Codex raised pass rate from 71.6% to 89.7%, while Codex reviewing Claude lowered it from 91.4% to 82.8%.

Reference graph

Works this paper leans on

36 extracted references · 1 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Evaluating Large Language Models Trained on Code

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Ger- stein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large lan- guage models. InProceedings of the 2024 Confer- ence of the North American Chapter of the Associ- ation for Computational Lin...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Read the full problem statement, constraints, and template before writing code

  3. [3]

    Produce the best practical algorithm for the stated constraints, aiming for the optimal time complexity when possible

  4. [4]

    Do not return a brute-force, quadratic, exponential, or placeholder solution when the constraints require a more efficient algorithm

  5. [5]

    Use the required method signature from the template exactly

  6. [6]

    Ensure the algorithm is complete and submission-ready, not a sketch or partial attempt

  7. [7]

    Handle edge cases implied by the statement and constraints

  8. [8]

    </instructions> <format_requirements> Return plain Python source code only

    Return one complete Python solution that can be submitted directly. </instructions> <format_requirements> Return plain Python source code only. Start directly with the code. Do not include explanations, markdown fences, or surrounding commentary. The code must be valid, runnable Python 3 with no syntax errors. The code must fit the provided code template ...

  9. [9]

    Infer an algorithm that matches the constraints before writing code

  10. [10]

    Prefer the optimal or near-optimal solution for the input limits

  11. [11]

    Do not output brute-force code unless the constraints clearly make it acceptable

  12. [12]

    Use the exact method signature from the template

  13. [13]

    </instructions> <format_requirements> Return plain Python source code only

    Return only final submission code. </instructions> <format_requirements> Return plain Python source code only. Start directly with the code. Do not include explanations, markdown fences, or extra text. The code must be valid, runnable Python 3 with no syntax errors. The code must fit the provided code template without requiring manual edits. The solution ...

    2026

  14. [14]

    Limitations

    Did the paper include a section about lim- itations? Yes.The paper contains a ded- icated “Limitations” section that discusses benchmark size, contamination risk, API non- determinism, metric compression, o4-mini’s undocumented decoding regime, prompt con- trast limitations, and scope restrictions

  15. [15]

    Ethics Statement

    Did the paper include a section about eth- ical considerations? Yes.The paper con- tains a dedicated “Ethics Statement” section addressing platform-use norms, API terms of service, dual-use concerns around automated code generation, and the absence of human subjects or personal data

  16. [16]

    Did you describe the limitations of your work? Yes.Six distinct limitations are enu- merated in the Limitations section: bench- mark scope ( N= 100 , one language), con- tamination risk, narrow experimental protocol, API opacity, metric compression (binary pass/- fail), and o4-mini’s uncontrolled decoding

  17. [17]

    Did you discuss any potential negative so- cietal impacts of your work? Yes.The Ethics Statement acknowledges that improved repeated-run reliability could lower barriers to generating code in settings lacking adequate security review, and frames the contribution as an evaluation methodology rather than an endorsement of autonomous deployment

  18. [18]

    Artifacts

    Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes. Artifacts

  19. [19]

    Ex- perimental Setup

    Did you describe the study design, the cre- ation of the evaluation benchmark, and the data collection process? Yes.Section 3 (“Ex- perimental Setup”) describes problem selec- tion by recency rank, difficulty distribution, grading criteria, model families, decoding pa- rameters, prompt templates, normalization, and execution pipeline

  20. [20]

    The evaluation benchmark consists of the 100 most recently published LeetCode problems at collection time

    Did you use existing data or create new data? Yes(existing data). The evaluation benchmark consists of the 100 most recently published LeetCode problems at collection time. No new dataset was created; the prob- lems and test cases are sourced from the Leet- Code platform

  21. [21]

    No personal identi- 12 fiers or user-generated content beyond algo- rithmic problem statements are used

    Did you discuss whether and how con- sent was obtained from people whose data you’re using, if it applies? N/A.The prob- lems are algorithmically specified tasks con- taining no personal data. No personal identi- 12 fiers or user-generated content beyond algo- rithmic problem statements are used

  22. [22]

    consist[s] of algorithmic specifications only and con- tain[s] no personally identifying information

    Did you discuss whether the data you are using/curating contains personally identi- fiable information (PII)? Yes.Section 3 ex- plicitly states that the benchmark “consist[s] of algorithmic specifications only and con- tain[s] no personally identifying information.”

  23. [23]

    Model snapshots (provider, API identifier, access date) are listed in Appendix B

    Did you provide documentation of the ar- tifacts, e.g., coverage of domains and lan- guages, demographic groups represented, etc.? Yes.Section 3 documents bench- mark composition: 100 problems, three diffi- culty tiers (20 Easy / 50 Medium / 30 Hard), algorithmic topics, Python-only evaluation, and selection methodology. Model snapshots (provider, API ide...

  24. [24]

    Per-model result dis- tributions appear in Table 1, Figure 2, and Figure 3

    Did you report the basic statistics of the data you used? Yes.Benchmark statistics (size, difficulty split, topic coverage) are re- ported in Section 3.1. Per-model result dis- tributions appear in Table 1, Figure 2, and Figure 3. Computational experiments

  25. [25]

    The paper reports the API identi- fiers, model tier designations, and access dates in Table 1 and Appendix B as the available model descriptors

    If you ran experiments, did you report the number of parameters for each model? No.Parameter counts for the evaluated mod- els are not publicly disclosed by all provider families (OpenAI, Anthropic, Google, Qwen, DeepSeek). The paper reports the API identi- fiers, model tier designations, and access dates in Table 1 and Appendix B as the available model d...

  26. [26]

    Section 3.4 specifies temperature T= 0.3 , top-p= 0.9 , R= 5 runs per problem, and the default 4,096-token output budget

    If you ran experiments, did you report the hyperparameters used and the range of hyperparameters you searched over? Yes. Section 3.4 specifies temperature T= 0.3 , top-p= 0.9 , R= 5 runs per problem, and the default 4,096-token output budget. Reason- ing model budget exceptions are documented. The study does not perform hyperparameter search; values are f...

  27. [27]

    Did you use the same evaluation procedure for all experiments? Yes.All 32 mod- el/prompt configurations are evaluated on the identical 100-problem set using the same de- terministic judge, normalization pipeline, and metric definitions (RLPR, PSR, A V)

  28. [28]

    API-based inference costs depend on provider pricing and token consumption, which vary per model and are not uniformly available

    Did you report the total amount of com- pute used to run your experiments? No. API-based inference costs depend on provider pricing and token consumption, which vary per model and are not uniformly available. The paper reports the number of evaluation instances (16,000 total) and the per-model output token budgets as proxies for compute scale. Exact compu...

  29. [29]

    The judge is a deterministic code execution har- ness; no human annotation is used

    Did you use human annotators (e.g., crowd- workers) or research participants in any way? No.All evaluation is automated. The judge is a deterministic code execution har- ness; no human annotation is used

  30. [30]

    Did you report the instructions given to an- notators and their payment/compensation? N/A.No human annotators were used

  31. [31]

    Model development (if applicable)

    Did you discuss potential biases in the hu- man annotation process and/or possible mitigation strategies? N/A.No human anno- tation was used. Model development (if applicable)

  32. [32]

    Did you report model validation results? N/A.The paper does not train or fine-tune models; it evaluates pre-existing commercial models through provider APIs

  33. [33]

    MINIMAL) is reported in Section 4.5 with paired McNemar tests

    Did you report the results of any abla- tion study? Yes.A controlled prompt ab- lation (DETAILEDvs. MINIMAL) is reported in Section 4.5 with paired McNemar tests. A reasoning-vs.-standard architecture compari- son is reported in Section 4.4. Both analy- ses are explicitly scoped as exploratory given sample-size constraints

  34. [34]

    Computational social science / Information ex- traction

    Did you use the same evaluation procedure for all model variants? Yes.Identical met- rics, dataset, judge, normalization, and sta- tistical inference procedures are applied to all 32 model/prompt configurations (with the 13 documented exception that o4-mini does not expose temperature/top-pcontrols). Computational social science / Information ex- traction

  35. [35]

    If your work is on applications in the so- cial sciences or other fields, did you discuss domain expertise? N/A.The paper is an em- pirical NLP/ML evaluation study, not a social science or information-extraction application

  36. [36]

    The Ethics Statement explicitly cautions against using improved coding per- formance as justification for fully autonomous code deployment without security review

    Did you describe potential applications and use cases of your work in a respon- sible manner? Yes.The Conclusion frames the metrics as evaluation tools for reproducibility-sensitive software engineering workflows. The Ethics Statement explicitly cautions against using improved coding per- formance as justification for fully autonomous code deployment with...