{"id":"cff0cb50-1446-4e08-b1c6-f04412b83d01","arxiv_id":"2412.15118","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A tree-based, inference-only framework combining execution metrics with LLM self-critique improves code generation correctness and efficiency across models and benchmarks.","lead":"This paper introduces ORPS, a search framework that lets large language models critique and refine their own code using execution feedback, without training a separate reward model. On five models and three coding benchmarks, it reports large gains in correctness and efficiency, especially on complex problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The advertised 26.9% Pass@1 gain mixes ORPS (w/T) — which scores search nodes with dataset-provided gold tests — against baselines without that test access, so the main quantitative claim is not yet cleanly supported.","rationale":"The reader's weakest assumption concerns the undisclosed outcome-reward weights and normalization in Section 3.2. That is a real reproducibility concern, but I see a more load-bearing problem in the central quantitative claim: the 26.9% Pass@1 improvement and 42.2% runtime reduction are not defined by setting, and Table 2 mixes ORPS (w/T) rows — which receive dataset-provided gold tests at search time — against baselines that do not receive those tests. Since the same tests also define final Pass@1, the w/T rows are not comparable to non-w/T CoT, Reflexion, or BoN entries, and the LDB (w/T) baseline is the only matched comparison. The PRM-elimination argument in Table 5 controls LLM calls and does not rely on gold tests, so it is the strongest evidence for the paper's broader thesis. The paper is honest about labeling the w/T rows, but the headline averages obscure the distinction. The reader noted the w/T issue in the rationale, but did not make it the central concern; I therefore partially agree with the reader's weakest_assumption. The verdict should remain CONDITIONAL: the core idea is plausible and the same-call PRM comparison is informative, but the main advertised numbers need to be recomputed and reported separately by test-access condition before the claim that ORPS eliminates trained PRMs while improving correctness by 26.9% can be accepted as stated.","tokens_in":24968,"tokens_out":8675,"duration_ms":78177,"concrete_test":"Recompute the advertised 26.9% Pass@1 and 42.2% runtime improvements from Table 2 in three partitions: (i) ORPS without gold tests vs CoT, Reflexion, and BoN; (ii) ORPS (w/T) vs LDB (w/T); and (iii) ORPS (w/T) vs the non-w/T baselines. Report per-partition averages across all 5 models and 3 datasets. If the headline 26.9% is dominated by partition (iii), revise the abstract and contributions to state that the headline gain requires gold-test access, and restrict the PRM-elimination claim to the controlled same-call comparisons in Table 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines 'w/ T' as methods using dataset-provided unit tests, and Table 2 marks only LDB among baselines with (w/ T); CoT, Reflexion, and BoN are evaluated without those tests. ORPS (w/ T) uses the same unit tests that define Pass@1 both to score each search node (Section 3.2) and to decide final pass/fail. This makes the w/ T rows an evaluation-leakage condition relative to the non-w/ T baselines, not a fair comparison. The abstract and contributions report an average 'Pass@1 improvement of 26.9%' and '42.2%' runtime reduction without stating whether these aggregates include the w/ T rows. Recomputing Table 2 excluding the w/ T rows shows materially smaller gains, and on MBPP ORPS without gold tests is frequently below BoN, as the paper itself acknowledges. The w/ T setting is a legitimate ablation for studying test quality, but it cannot support the blanket claim that inference-only ORPS improves correctness by 26.9% over methods that do not receive the same test access. To sustain the central claim, the authors must report per-setting averages and compare ORPS (w/ T) only against baselines given the same unit tests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Outcome-Refining Process Supervision (ORPS), an inference-only tree-search framework for code generation. At each beam-search step, an LLM generates candidate reasoning-and-code refinements, which are executed and profiled to produce a weighted outcome reward from static and dynamic metrics; the same LLM then self-critiques the candidate and emits a process reward, and the two scores are combined to select the beam. Experiments on LBPP, HumanEval, and MBPP with five base models report average Pass@1 gains of 26.9% and runtime reductions of 42.2% over baselines such as CoT, Reflexion, LDB, and Best-of-N, plus additional experiments with trained PRMs and a CodeContests extension.","tokens_in":25284,"tokens_out":3742,"duration_ms":32022,"significance":"If the central claim holds, the paper would make a useful contribution: it demonstrates that execution-grounded LLM self-critique plus beam search can replace specially trained Process Reward Models in code generation, with open-sourced code and controlled call-budget experiments. The multi-metric outcome reward and the PRM comparison are interesting beyond the specific benchmarks. However, the headline quantitative claim currently mixes settings with and without access to dataset-provided unit tests, and the outcome-reward construction is underspecified, so the significance is real but the main comparison needs to be re-reported before the claim is supportable.","major_comments":[{"comment":"The headline claim of an average Pass@1 improvement of 26.9% and a 42.2% runtime reduction is not supported by a clean comparison. In Table 2, the ORPS (w/ T) rows use dataset-provided unit tests both to score every search node (Section 3.2) and to determine final pass/fail, whereas CoT, Reflexion, and Best-of-N are evaluated without those tests (only LDB is marked w/ T). Aggregating the w/ T rows into the same average as the no-test baselines therefore conflates an evaluation-leakage condition with a fair inference-only comparison. The authors should report the averages separately for the w/ T and no-test settings, state which setting each advertised aggregate refers to, and compare ORPS (w/ T) only against baselines given the same unit-test access.","section":"Abstract and Section 4.2, Table 2"},{"comment":"The outcome reward is defined as omega_t = sum_{k=1}^M beta_k * normalize(m_k), but the paper never reports beta_k, the normalization method, or the range over which normalization is computed (per candidate, per problem, or per benchmark). Since the outcome reward directly controls beam selection and the claimed efficiency gains are partly driven by including execution-speed metrics, the missing specification makes the central mechanism irreproducible and the results non-robust to plausible alternative weightings. The paper should provide the exact weights, the normalization formula, and a sensitivity analysis over these choices.","section":"Section 3.2, Eq. (1)"},{"comment":"The computational-efficiency claim is not fully supported because the hyperparameters used at each call budget are not reported. Algorithm 1 and Section 4.5 give ORPS a cost of 2*N*(K*T+1) LLM calls, but with the default N=20, K=3, T=5 from Appendix A this would be 640 calls, not the 20, 50, or 100 calls listed in Table 5. The authors must state the (N,K,T) settings used for each call limit and, ideally, also report the resulting beam size and search depth, since otherwise the comparison to Reflexion, LDB, and REx at equal call counts is not well defined.","section":"Section 4.5, Table 5 and Algorithm 1"},{"comment":"The claim of 'consistent gains' across benchmarks is contradicted by the MBPP rows without test access: ORPS is below Best-of-N for Llama-3.1-8B (71.8 vs 73.5), DeepSeek-Coder-7B (73.2 vs 74.3), Qwen-2.5-Coder-7B (76.7 vs 82.9), and Qwen-2.5-Coder-14B (76.3 vs 81.7). The text acknowledges only a 'slight disadvantage' and attributes it to benchmark simplicity and contamination, but this does not explain why a method with strategic exploration underperforms simple sampling on those rows. The authors should present per-setting averages and discuss MBPP explicitly if they retain the 'consistent gains' language in the abstract.","section":"Section 4.2, MBPP results"}],"minor_comments":[{"comment":"The word 'texual' should be 'textual'.","section":"Section 3.3"},{"comment":"The call-budget comparison counts only inference calls for trained PRMs and excludes their training data and compute; the paper should state this limitation explicitly when claiming that ORPS 'eliminates' PRM training overhead.","section":"Section 4.5"},{"comment":"The y-axis of Figure 4 is labeled 'Pass@1' with values around 0.3-0.6, while Table 2 reports percentages; please make the units consistent.","section":"Figure 4"},{"comment":"The hyperparameter appendix lists alpha and beta but not beta_k or the normalization scheme for Eq. (1); this should be added for reproducibility.","section":"Appendix A"},{"comment":"The comparison between 'Outcome ✓' and 'Line ✓' PRMs is interesting, but the training data sizes and validation procedure for the PRMs are described only briefly; a reference to the full training configuration in Appendix A would help.","section":"Section 4.4 and Table 4"},{"comment":"In the case study, the 'improved' solution still uses a nested loop and the critic's suggested O(n log n) approach is not actually implemented, so the example does not fully illustrate the claimed algorithmic-level improvement.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable if the evaluation is re-segmented by unit-test access. The current abstract overstates the result by pooling w/ T and no-test rows. The missing beta_k specification is the kind of detail that a careful reader will treat as a reproducibility red flag, so it must be addressed before acceptance. The CodeContests appendix is a good addition but shares the same call-budget and hyperparameter transparency issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core of this paper is worth your attention. ORPS is a clean inference-only framework: tree-structured beam search over reasoning and code states, execution profiling for outcome rewards, and LLM self-critique for process rewards. It does a real job of showing that trained PRMs are not necessary for code generation when you have executable feedback. The ablations are thoughtful (removing execution, removing reasoning, single-metric optimization), the code is open-sourced, and the controlled LLM-call comparison against trained PRMs is a fair and useful test. That part largely holds up.\n\nThe problems are with the presentation, not the method's bones. First and most important: the abstract and contributions advertise an average 26.9% Pass@1 improvement and 42.2% runtime reduction without separating the w/ T rows. In Table 2, ORPS (w/ T) uses dataset-provided gold tests to score every search node, while CoT, Reflexion, and BoN get no such access. LDB is the only baseline also marked w/ T. The aggregate claim therefore mixes an apples-to-oranges comparison. When you look only at the no-gold-test rows, ORPS is often at or below BoN on HumanEval and MBPP, and sometimes on LBPP too. The w/ T condition is a legitimate ablation, but it cannot support the blanket statement that ORPS beats outcome-supervision baselines by 26.9%. The paper should report per-setting averages and compare ORPS (w/ T) only against LDB (w/ T), and ORPS without gold tests against the other baselines.\n\nThe second issue is reproducibility: the outcome reward in Section 3.2 is a weighted sum of normalized metrics, but the weights beta_k and the normalization method are never disclosed. Appendix A gives alpha=0.5 and beta=0.5 but not the metric-level weights. That matters because the search's guidance depends directly on those choices. Also, no error bars anywhere, and on a 162-problem benchmark, differences of a few points are within noise.\n\nThese are fixable. The framework itself is coherent, the PRM comparison is honest, and the authors already acknowledge the MBPP weakness and test contamination. This paper deserves a serious referee, but the revision must separate the w/ T results from the no-gold-test results in every aggregate claim, report the outcome reward weights, and ideally add variance estimates. I'd take it for review, not desk-reject.","headline":"The ORPS framework is a genuine and useful contribution, but the headline 26.9% gain mixes gold-test-guided search against baselines that don't get gold tests, so the flagship claim is overstated as presented.","tokens_in":25770,"tokens_out":2972,"would_cite":true,"duration_ms":27589,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ORPS, a search framework that executes and profiles each code candidate and uses self-critique to score the reasoning behind it, claims that specially trained process reward models are unnecessary for code generation, reporting 26.9%…","keywords":["code generation","process reward model","outcome supervision","tree search","execution feedback","self-critique","Pass@1","ORPS"],"falsifier":"Re-run ORPS on LBPP at a fixed LLM-call budget (say 100 calls) while randomly permuting or uniformizing the per-metric weights $\\beta_k$ in the outcome reward. If Pass@1 stays near 64.2, the un-reported weights are not load-bearing; if it collapses, the reported gains depend on exactly those choices.","tokens_in":24803,"feed_emoji":"⚡","tokens_out":8300,"duration_ms":53606,"temperature":0.7,"pith_summary":"Outcome Refining Process Supervision (ORPS) is a search framework that tries to settle a practical question in code generation: do we need to train a separate process reward model (PRM) to guide an LLM's intermediate reasoning, or can verifiable execution feedback do the job? The paper's answer is the latter. ORPS runs a beam search over reasoning chains and code candidates, executes and profiles each candidate, and asks the same LLM to critique the result and assign a process reward; the combined step score steers the search toward better algorithmic strategies. Across five LLMs and three benchmarks the authors report an average Pass@1 improvement of 26.9% and a 42.2% reduction in solution runtime, and they show that trained PRMs—even with human-filtered data—underperform this inference-only alternative. If correct, the result would let practitioners improve code correctness and efficiency without costly reward-model training.","feed_headline":"Execution-guided search tops trained reward models for code","feed_subtitle":"ORPS reports 26.9% higher Pass@1 and 42.2% faster code across five models and three benchmarks.","key_machinery":"The load-bearing mechanism is a beam search over structured states whose expansion is scored by a weighted blend of an outcome reward and a process reward. The outcome reward $\\omega_t = \\sum_{k=1}^M \\beta_k \\cdot \\mathrm{normalize}(m_k)$ aggregates dynamic execution metrics (correctness, time, CPU instructions, page faults) and static complexity metrics (code length, AST node count, cyclomatic and cognitive complexity). The process reward $\\rho_t$ is written by the same LLM acting as critic, who reads the reasoning chain, code, and execution profile and produces both a textual critique and a numeric score. Combining them via $q_t = \\alpha \\rho_t + \\beta \\omega_t$ with $\\alpha + \\beta = 1$ makes execution feedback anchor the LLM's subjective judgment, preventing reward hacking; setting $\\beta = 0$ recovers pure process supervision and $\\alpha = 0$ recovers outcome supervision, so the framework claims to unify the two.","core_discovery":"On ORPS's own terms, process and outcome supervision are not separate methods but two endpoints of a single spectrum, and execution outcomes should be folded into process rewards. The framework represents each search state as a tuple of reasoning chain, code, execution feedback, outcome reward, critic text, and process reward. After the LLM proposes candidate refinements, the code is executed and profiled on correctness, time, CPU instructions, page faults, and static complexity measures; those normalized metrics form the outcome reward. The LLM then produces a textual critique and a numerical process reward, and beam search keeps the top states by $q_t = \\alpha \\rho_t + \\beta \\omega_t$. The paper's strongest empirical claim is that with dataset-provided unit tests, ORPS reaches 91.4% Pass@1 on HumanEval and 67.1% on LBPP with Llama-3.1-8B, and a 7B model under ORPS can outperform a 14B model without it, which the authors take as evidence that reasoning space matters more than model scale.","pith_inferences":["The paper leaves open the sensitivity of its speed gains to the unreported per-metric weights $\\beta_k$: if those weights are arbitrary, the 42.2% runtime reduction may partly reflect a built-in preference for fast code rather than a discovered algorithmic insight.","A direct test of the claimed mechanism would randomize or uniformize the $\\beta_k$ weights at a fixed call budget; if Pass@1 barely moves, the correctness gains come from search structure itself, while if it collapses, the reported results depend on exactly those choices.","The same tree-search design could transfer to other verifiable domains—formal theorem proving, SQL query generation, or robot control in a simulator—where an executor can score candidates and an LLM critic can judge intermediate reasoning, so the claim that verifiable outcomes can replace learned PRMs is likely broader than code.","The paper's own tables suggest that access to gold unit tests is a major driver of the largest gains, so the practical benefit on new, contamination-free tasks with only self-generated tests is probably noticeably smaller than the headline 26.9% improvement."],"forward_implications":["If the central claim is right, trained PRMs can be dropped from code-generation pipelines without losing search guidance, because execution feedback plus self-critique supplies the same steering signal more cheaply.","Smaller models can become competitive with larger ones when given a structured search space, implying that inference-time reasoning budget is a substitute for parameter count.","Execution profiling can pull solutions out of local optima: code that passes all tests but runs in $O(n^2)$ is scored down on complexity metrics and pushed toward better algorithms, not just repaired.","The $\\alpha/\\beta$ framing predicts intermediate supervision regimes between pure outcome and pure process supervision, so practitioners can tune toward correctness or efficiency by adjusting weights.","Under equal LLM-call budgets, ORPS scales better than Best-of-N and repair-based methods, meaning the reported improvements are attributed to search strategy rather than brute-force sampling."],"supporting_citations":[{"why":"Defines the Reflexion execution-feedback baseline that ORPS must beat and represents the local-repair paradigm ORPS claims to supersede.","marker":"Shinn et al., 2024"},{"why":"Supplies the LDB debugger-driven baseline, including its use of dataset-provided tests, against which ORPS compares in the main results.","marker":"Zhong et al., 2024"},{"why":"Introduces process reward models and step-by-step verification, the training-heavy paradigm ORPS argues is unnecessary.","marker":"Lightman et al., 2023"},{"why":"Shows that self-repair and reasoning quality correlate with code generation outcomes, motivating tree-structured exploration of solutions.","marker":"Olausson et al., 2023"},{"why":"Provides REx, the repair-based baseline that most directly frames code repair as an exploration-exploitation tradeoff and shows diminishing returns against ORPS in cost-controlled runs.","marker":"Tang et al., 2024a"},{"why":"Supplies the automated process supervision approach adapted into the trained PRM-GPT and PRM-Human comparison models.","marker":"Luo et al., 2024"},{"why":"Introduces the LBPP benchmark and reports contamination rates, providing the hard, recent dataset where ORPS shows its largest gains.","marker":"Matton et al., 2024"},{"why":"Defines HumanEval and the test-case-based outcome supervision standard, one of the three evaluation benchmarks.","marker":"Chen et al., 2021b"},{"why":"Supplies evidence that LLMs struggle to predict execution outcomes, supporting the paper's motivation for grounding search in executable verification.","marker":"Jain et al., 2024"}],"fun_headline_variants":["Execution feedback unifies process and outcome rewards for code","ORPS: Tree search with execution metrics beats trained reward models","Self-critique with runtime feedback boosts code generation","Unifying process and outcome rewards via executable verification","Execution-guided search outperforms trained reward models for code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"ORPS assumes that the hand-picked weighted combination of execution and complexity metrics—whose weights and normalization the paper never reports—reliably ranks candidate solutions by quality, so that the search is steered toward genuinely better code.","fun_headline_variants_meta":{"raw":{"variants":["Execution feedback unifies process and outcome rewards for code","ORPS: Tree search with execution metrics beats trained reward models","Self-critique with runtime feedback boosts code generation","Unifying process and outcome rewards via executable verification","Execution-guided search outperforms trained reward models for code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2316,"prompt_tokens":925,"completion_tokens":1391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1314}},"tokens_in":541,"tokens_out":1391,"duration_ms":6653,"temperature":1.0,"reasoning_tokens":1314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:35:52.040975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run ORPS on LBPP at a fixed LLM-call budget (say 100 calls) while randomly permuting or uniformizing the per-metric weights $\\beta_k$ in the outcome reward. If Pass@1 stays near 64.2, the un-reported weights are not load-bearing; if it collapses, the reported gains depend on exactly those choices.","supporting_citations":[],"review_version":1}