Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ORPS, a search framework that executes and profiles each code candidate and uses self-critique to score the reasoning behind it, claims that specially trained process reward models are unnecessary for code generation, reporting 26.9%…

desk verdict The ORPS framework is a genuine and useful contribution, but the headline 26.9% gain mixes gold-test-guided search against baselines that don't get gold tests, so the flagship claim is overstated as presented. read the letter →

arxiv 2412.15118 v2 pith:55TREVZF submitted 2024-12-19 cs.CL cs.AIcs.LGcs.SE

Zhuohao Yu , Weizheng Gu , Yidong Wang , Xingru Jiang , Zhengran Zeng , Jindong Wang , Wei Ye , Shikun Zhang This is my paper · ORCID
classification cs.CLcs.AIcs.LGcs.SE
keywords codegenerationprocessrewardmodeloutcomesupervisiontreesearchexecutionfeedbackself-critiquePass@1ORPS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Outcome Refining Process Supervision (ORPS) is a search framework that tries to settle a practical question in code generation: do we need to train a separate process reward model (PRM) to guide an LLM's intermediate reasoning, or can verifiable execution feedback do the job? The paper's answer is the latter. ORPS runs a beam search over reasoning chains and code candidates, executes and profiles each candidate, and asks the same LLM to critique the result and assign a process reward; the combined step score steers the search toward better algorithmic strategies. Across five LLMs and three benchmarks the authors report an average Pass@1 improvement of 26.9% and a 42.2% reduction in solution runtime, and they show that trained PRMs—even with human-filtered data—underperform this inference-only alternative. If correct, the result would let practitioners improve code correctness and efficiency without costly reward-model training.

What carries the argument

The load-bearing mechanism is a beam search over structured states whose expansion is scored by a weighted blend of an outcome reward and a process reward. The outcome reward $\omega_t = \sum_{k=1}^M \beta_k \cdot \mathrm{normalize}(m_k)$ aggregates dynamic execution metrics (correctness, time, CPU instructions, page faults) and static complexity metrics (code length, AST node count, cyclomatic and cognitive complexity). The process reward $\rho_t$ is written by the same LLM acting as critic, who reads the reasoning chain, code, and execution profile and produces both a textual critique and a numeric score. Combining them via $q_t = \alpha \rho_t + \beta \omega_t$ with $\alpha + \beta = 1$ makes execution feedback anchor the LLM's subjective judgment, preventing reward hacking; setting $\beta = 0$ recovers pure process supervision and $\alpha = 0$ recovers outcome supervision, so the framework claims to unify the two.

What would settle it

Re-run ORPS on LBPP at a fixed LLM-call budget (say 100 calls) while randomly permuting or uniformizing the per-metric weights $\beta_k$ in the outcome reward. If Pass@1 stays near 64.2, the un-reported weights are not load-bearing; if it collapses, the reported gains depend on exactly those choices.

Watch

Extended reading notes

Core claim

On ORPS's own terms, process and outcome supervision are not separate methods but two endpoints of a single spectrum, and execution outcomes should be folded into process rewards. The framework represents each search state as a tuple of reasoning chain, code, execution feedback, outcome reward, critic text, and process reward. After the LLM proposes candidate refinements, the code is executed and profiled on correctness, time, CPU instructions, page faults, and static complexity measures; those normalized metrics form the outcome reward. The LLM then produces a textual critique and a numerical process reward, and beam search keeps the top states by $q_t = \alpha \rho_t + \beta \omega_t$. The paper's strongest empirical claim is that with dataset-provided unit tests, ORPS reaches 91.4% Pass@1 on HumanEval and 67.1% on LBPP with Llama-3.1-8B, and a 7B model under ORPS can outperform a 14B model without it, which the authors take as evidence that reasoning space matters more than model scale.

Load-bearing premise

ORPS assumes that the hand-picked weighted combination of execution and complexity metrics—whose weights and normalization the paper never reports—reliably ranks candidate solutions by quality, so that the search is steered toward genuinely better code.

Editorial extensions

If this is right

  • If the central claim is right, trained PRMs can be dropped from code-generation pipelines without losing search guidance, because execution feedback plus self-critique supplies the same steering signal more cheaply.
  • Smaller models can become competitive with larger ones when given a structured search space, implying that inference-time reasoning budget is a substitute for parameter count.
  • Execution profiling can pull solutions out of local optima: code that passes all tests but runs in $O(n^2)$ is scored down on complexity metrics and pushed toward better algorithms, not just repaired.
  • The $\alpha/\beta$ framing predicts intermediate supervision regimes between pure outcome and pure process supervision, so practitioners can tune toward correctness or efficiency by adjusting weights.
  • Under equal LLM-call budgets, ORPS scales better than Best-of-N and repair-based methods, meaning the reported improvements are attributed to search strategy rather than brute-force sampling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open the sensitivity of its speed gains to the unreported per-metric weights $\beta_k$: if those weights are arbitrary, the 42.2% runtime reduction may partly reflect a built-in preference for fast code rather than a discovered algorithmic insight.
  • A direct test of the claimed mechanism would randomize or uniformize the $\beta_k$ weights at a fixed call budget; if Pass@1 barely moves, the correctness gains come from search structure itself, while if it collapses, the reported results depend on exactly those choices.
  • The same tree-search design could transfer to other verifiable domains—formal theorem proving, SQL query generation, or robot control in a simulator—where an executor can score candidates and an LLM critic can judge intermediate reasoning, so the claim that verifiable outcomes can replace learned PRMs is likely broader than code.
  • The paper's own tables suggest that access to gold unit tests is a major driver of the largest gains, so the practical benefit on new, contamination-free tasks with only self-generated tests is probably noticeably smaller than the headline 26.9% improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Outcome-Refining Process Supervision (ORPS), an inference-only tree-search framework for code generation. At each beam-search step, an LLM generates candidate reasoning-and-code refinements, which are executed and profiled to produce a weighted outcome reward from static and dynamic metrics; the same LLM then self-critiques the candidate and emits a process reward, and the two scores are combined to select the beam. Experiments on LBPP, HumanEval, and MBPP with five base models report average Pass@1 gains of 26.9% and runtime reductions of 42.2% over baselines such as CoT, Reflexion, LDB, and Best-of-N, plus additional experiments with trained PRMs and a CodeContests extension.

Significance. If the central claim holds, the paper would make a useful contribution: it demonstrates that execution-grounded LLM self-critique plus beam search can replace specially trained Process Reward Models in code generation, with open-sourced code and controlled call-budget experiments. The multi-metric outcome reward and the PRM comparison are interesting beyond the specific benchmarks. However, the headline quantitative claim currently mixes settings with and without access to dataset-provided unit tests, and the outcome-reward construction is underspecified, so the significance is real but the main comparison needs to be re-reported before the claim is supportable.

major comments (4)
  1. [Abstract and Section 4.2, Table 2] The headline claim of an average Pass@1 improvement of 26.9% and a 42.2% runtime reduction is not supported by a clean comparison. In Table 2, the ORPS (w/ T) rows use dataset-provided unit tests both to score every search node (Section 3.2) and to determine final pass/fail, whereas CoT, Reflexion, and Best-of-N are evaluated without those tests (only LDB is marked w/ T). Aggregating the w/ T rows into the same average as the no-test baselines therefore conflates an evaluation-leakage condition with a fair inference-only comparison. The authors should report the averages separately for the w/ T and no-test settings, state which setting each advertised aggregate refers to, and compare ORPS (w/ T) only against baselines given the same unit-test access.
  2. [Section 3.2, Eq. (1)] The outcome reward is defined as omega_t = sum_{k=1}^M beta_k * normalize(m_k), but the paper never reports beta_k, the normalization method, or the range over which normalization is computed (per candidate, per problem, or per benchmark). Since the outcome reward directly controls beam selection and the claimed efficiency gains are partly driven by including execution-speed metrics, the missing specification makes the central mechanism irreproducible and the results non-robust to plausible alternative weightings. The paper should provide the exact weights, the normalization formula, and a sensitivity analysis over these choices.
  3. [Section 4.5, Table 5 and Algorithm 1] The computational-efficiency claim is not fully supported because the hyperparameters used at each call budget are not reported. Algorithm 1 and Section 4.5 give ORPS a cost of 2*N*(K*T+1) LLM calls, but with the default N=20, K=3, T=5 from Appendix A this would be 640 calls, not the 20, 50, or 100 calls listed in Table 5. The authors must state the (N,K,T) settings used for each call limit and, ideally, also report the resulting beam size and search depth, since otherwise the comparison to Reflexion, LDB, and REx at equal call counts is not well defined.
  4. [Section 4.2, MBPP results] The claim of 'consistent gains' across benchmarks is contradicted by the MBPP rows without test access: ORPS is below Best-of-N for Llama-3.1-8B (71.8 vs 73.5), DeepSeek-Coder-7B (73.2 vs 74.3), Qwen-2.5-Coder-7B (76.7 vs 82.9), and Qwen-2.5-Coder-14B (76.3 vs 81.7). The text acknowledges only a 'slight disadvantage' and attributes it to benchmark simplicity and contamination, but this does not explain why a method with strategic exploration underperforms simple sampling on those rows. The authors should present per-setting averages and discuss MBPP explicitly if they retain the 'consistent gains' language in the abstract.
minor comments (6)
  1. [Section 3.3] The word 'texual' should be 'textual'.
  2. [Section 4.5] The call-budget comparison counts only inference calls for trained PRMs and excludes their training data and compute; the paper should state this limitation explicitly when claiming that ORPS 'eliminates' PRM training overhead.
  3. [Figure 4] The y-axis of Figure 4 is labeled 'Pass@1' with values around 0.3-0.6, while Table 2 reports percentages; please make the units consistent.
  4. [Appendix A] The hyperparameter appendix lists alpha and beta but not beta_k or the normalization scheme for Eq. (1); this should be added for reproducibility.
  5. [Section 4.4 and Table 4] The comparison between 'Outcome ✓' and 'Line ✓' PRMs is interesting, but the training data sizes and validation procedure for the PRMs are described only briefly; a reference to the full training configuration in Appendix A would help.
  6. [Appendix F] In the case study, the 'improved' solution still uses a nested loop and the critic's suggested O(n log n) approach is not actually implemented, so the example does not fully illustrate the claimed algorithmic-level improvement.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline 26.9% Pass@1 gain partly reduces by construction: ORPS (w/ T) scores search nodes with the same dataset unit tests used to define Pass@1, while the comparison baselines lack that test access.

  1. self definitional [Abstract; Section 1 Contributions; Section 3.2; Algorithm 1; Section 4.1; Table 2 caption]
    "Experiments across 5 models and 3 benchmarks show consistent gains, with 26.9% higher correctness and 42.2% improved code efficiency. ... U may be either generated by the LLM itself based on the problem or provided by the dataset. ... In Table 2, methods using unit tests from datasets are denoted by (w/ T). ... Pass@1: solutions passing all test cases."

    In the (w/ T) condition, the dataset-provided unit tests are used both to score search candidates and to compute the reported Pass@1. Algorithm 1 scores each candidate with 'feedback, outcome rew <- execute and profile(code, U)', and beam selection keeps the 'Top-K paths with highest step score', where the step score includes the outcome reward built from those same tests. Thus the ORPS (w/ T) rows in Table 2 are not independent predictions of Pass@1 on held-out tests; they are selections that directly optimize pass rate on the evaluation tests.

full rationale

ORPS's core search loop is not circular in the equation sense: the step score q_t = alpha * rho_t + beta * omega_t is computed from execution feedback and LLM self-critique, and the ablations (Table 3) plus controlled-call comparisons (Table 5) provide independent evidence that execution grounding and reasoning contribute. The PRM comparison is self-contained, and the self-citations (e.g., FreeEval) are tooling-related and not load-bearing. However, one load-bearing quantitative claim is partly self-definitional. In the (w/ T) condition, the dataset-provided unit tests are fed into the outcome reward via Algorithm 1, and the same tests define Pass@1 in Table 2. Beam selection then chooses states with the highest step score, i.e., it selects for passing those tests. Therefore the ORPS (w/ T) Pass@1 values are selection outcomes on the evaluation metric, not independent predictions. The abstract's unqualified 26.9% average mixes these rows with baselines that did not receive dataset test access, so the headline overstates what the reasoning framework itself contributes. The non-w/ T rows and controlled-call experiments remain non-circular, so this is partial circularity rather than a fully forced derivation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework does not introduce new physical entities. It relies on a set of hand-chosen weights and domain assumptions about the usefulness of LLM self-critique and static metrics, which are not fully justified and could affect the reported gains.

free parameters (5)
  • alpha (process reward weight) = 0.5
    Set by hand in Appendix A.1; balances process and outcome rewards in step score q_t = alpha * rho_t + beta * omega_t.
  • beta (outcome reward weight) = 0.5
    Set by hand in Appendix A.1.
  • beta_k outcome metric weights = unspecified
    In omega_t = sum beta_k * normalize(m_k), Section 3.2; not reported, yet determines the outcome reward.
  • metric normalization scheme = unspecified
    normalize() is not defined; different normalizations (min-max, z-score) change relative metric contributions.
  • search depth T, beam width K, candidates N = 5, 3, 20
    Hyperparameters in Appendix A.1; chosen without sensitivity analysis.
assumptions (4)
  • domain assumption LLM self-critique, when supplied with execution feedback, produces a reliable process reward for guiding search.
    Central to the method; Section 3.3. Prior work cited by the paper (Huang et al. 2023, Stechly et al. 2024) questions this without external grounding.
  • domain assumption Unit tests (self-generated or dataset-provided) provide a sufficient correctness signal.
    The outcome reward and Pass@1 depend on these tests; Section 3.2 and 4.1. The paper shows self-generated tests are weaker than gold tests.
  • domain assumption Static code metrics (AST nodes, cyclomatic complexity, cognitive complexity) correlate with code quality and should be minimized.
    Used in outcome reward; Table 6. The paper's own single-metric ablations (Appendix D) show optimizing these in isolation degrades other metrics.
  • ad hoc to paper A weighted linear combination of normalized metrics is a valid scalar reward.
    Equation (1); no justification for additivity or commensurability of metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation." pith.science (2026). https://pith.science/paper/55TREVZF

@misc{pith2026241215118,
  author       = {Pith},
  title        = {Pith review of: Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55TREVZF}},
  note         = {Machine review of arXiv:2412.15118}
}
read the original abstract

Large Language Models excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision relies on learned reward models requiring costly training data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps. We introduce Outcome Refining Process Supervision, which unifies process and outcome supervision by leveraging executable verification: a tree-structured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that integrate runtime feedback with reasoning. Experiments across 5 models and 3 benchmarks show consistent gains, with 26.9% higher correctness and 42.2% improved code efficiency. The results demonstrate that ORPS enables LLMs to overcome local optima in code generation, suggesting a promising direction for combining verifiable outcomes with structured reasoning to tackle complex challenges. We open-source at: https://github.com/zhuohaoyu/ORPS

Figures

Figures reproduced from arXiv: 2412.15118 by the authors.

Figure 1
Figure 1. Comparison of outcome and process supervision. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Outcome-Refining Process Supervision framework overview. A language model serves as both programmer and critic in a step-by-step reasoning process. Through beam search, the framework maintains multiple solution trajectories, where each state contains reasoning chains, code implementations, and step reward. This approach has been predominantly used in math rea￾soning tasks(Luo et al., 2024; Jiang et al., 2024a), but … view at source ↗
Figure 3
Figure 3. Multi-dimensional Performance Analysis. Met￾rics are normalized against the LBPP standard solutions (1.0×) and averaged across all backbone models. Higher values indicate better performance [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance vs. Inference Budget. The y-axis represents Pass@1 scores on LBPP. The x-axis represents the number of candidates generated during inference. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance by Problem Class. Top-20 problem classes in LBPP showing success rates and unsolved cases for ORPS vs baseline. 4.7. Case Studies We also analyzed the improvements of ORPS across dif￾ferent problem categories. As shown in [PITH_FULL_IMAGE:figures/full_fig_…
Figure 6
Figure 6. Figure 6: Impact of single-metric optimization on code quality. Each value represents the difference from the baseline. Optimizing for a single metric significantly improves performance in that dimension but leads to severe degradation in others. cyclomatic complexity encourages…
Figure 7
Figure 7. Figure 7: Impact of reasoning quality on code quality. Each metric is normalized, where higher values indicate better performance. The red bars represent the code quality when generated from higher-quality reasoning, while the blue bars correspond to lower-quality reasoning. The…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncertainty-aware Reward Design Process

    cs.LG 2025-07 conditional novelty 6.0 of 10

    URDP couples LLM-based reward component design with uncertainty-weighted Bayesian optimization, reporting better reward quality and efficiency than Eureka and Text2Reward on three benchmarks.

  2. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  3. Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.

  4. Tree-of-Code: A Tree-Structured Exploring Framework for End-to-End Code Generation and Execution in Complex Task Handling

    cs.SE 2024-12 conditional novelty 5.0 of 10

    Tree-of-Code improves code-based LLM agents by generating complete end-to-end programs as tree nodes, using execution success to self-filter and majority voting to pick the final answer.

Reference graph

Works this paper leans on

103 extracted references · 10 canonical work pages · cited by 4 Pith papers

  1. [1]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  2. [2]

    Devansh Arpit, Stanis aw Jastrz e bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017. A closer look at memorization in deep networks. In International conference on machine learning, pages 233--242. PMLR

    2017

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023

    arXiv 2023

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  6. [6]

    G Ann Campbell. 2018. Cognitive complexity-a new way of measuring understandability. SonarSource SA, page 10

    2018

  7. [7]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109

    arXiv 2023

  8. [8]

    Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021 a . Evaluating the rationales of amateur investors. In Proceedings of the Web Conference 2021, pages 3987--3998

    2021

Show all 103 references
  1. [9]

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024 a . Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669

    2024 arXiv

  2. [10]

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024 b . Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553

    2024 arXiv

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 b . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  4. [12]

    Xinyun Chen, Maxwell Lin, Nathanael Sch \"a rli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128

    2023 arXiv

  5. [13]

    Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao. 2024 c . Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition. arXiv preprint arXiv:2402.11452

    2024 arXiv

  6. [14]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023)

    2023

  7. [15]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  8. [16]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  9. [17]

    Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan. 2024. Process supervision-guided policy optimization for code generation. arXiv preprint arXiv:2410.17621

    2024 arXiv

  10. [18]

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305

    2020 arXiv

  11. [19]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  12. [20]

    Jonathan St BT Evans. 2003. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454--459

    2003

  13. [21]

    Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806

    2017 arXiv

  14. [22]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

    2023 arXiv

  15. [23]

    Dahl, Justin Gilmer, Christopher J

    Varun Godbole, George E. Dahl, Justin Gilmer, Christopher J. Shallue, and Zachary Nado. 2023. http://github.com/google-research/tuning_playbook Deep learning tuning playbook . Version 1.0

    2023

  16. [24]

    Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning, volume 1. MIT Press

    2016

  17. [25]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738

    2023 arXiv

  18. [26]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  19. [27]

    Mingqian He, Yongliang Shen, Wenqi Zhang, Zeqi Tan, and Weiming Lu. 2024 a . Advancing process verification for large language models via tree-based preference learning. arXiv preprint arXiv:2407.00390

    2024 arXiv

  20. [28]

    Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. 2024 b . Semi-supervised reward modeling via iterative self-training. arXiv preprint arXiv:2409.06903

    2024 arXiv

  21. [29]

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701

    2020 arXiv

  22. [30]

    Hinton, Simon Osindero, and Yee Whye Teh

    Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527--1554

    2006

  23. [31]

    Lynette Hirschman and Robert Gaizauskas. 2001. Natural language question answering: the view from here. natural language engineering, 7(4):275--300

    2001

  24. [32]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

    2021

  25. [33]

    Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. 2024. Are llm-based evaluators confusing nlg quality criteria? arXiv preprint arXiv:2402.12055

    2024 arXiv

  26. [34]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  27. [35]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  28. [36]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  29. [37]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  30. [38]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  31. [39]

    Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. 2024 a . Technical report: Enhancing llm reasoning with reward-guided tree search. arXiv preprint arXiv:2411.11694

    2024 arXiv

  32. [40]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024 b . A survey on large language models for code generation. arXiv preprint arXiv:2406.00515

    2024 arXiv

  33. [41]

    Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F Chen, and Shafiq Joty. 2024. Learning planning-based reasoning by trajectories collection and process reward synthesizing. arXiv preprint arXiv:2402.00658

    2024 arXiv

  34. [42]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770

    2023 arXiv

  35. [43]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  36. [44]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  37. [45]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  38. [46]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Syste...

    2022

  39. [47]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  40. [48]

    Yucheng Li. 2023. An open source data contamination report for llama series models. arXiv preprint arXiv:2310.17589

    2023 arXiv

  41. [49]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  42. [50]

    Yachuan Liu, Liang Chen, Jindong Wang, Qiaozhu Mei, and Xing Xie. 2023. Meta semantic template for evaluation of large language models. arXiv preprint arXiv:2310.01448

    2023 arXiv

  43. [51]

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592

    2024 arXiv

  44. [52]

    Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. 2023. Let's reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.10080

    2023 arXiv

  45. [53]

    Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gall \'e . 2024. On leakage of code generation evaluation datasets. arXiv preprint arXiv:2407.07565

    2024 arXiv

  46. [54]

    Marvin Mu \ n oz Bar \'o n, Marvin Wyrich, and Stefan Wagner. 2020. An empirical validation of cognitive complexity as a measure of source code understandability. In Proceedings of the 14th ACM/IEEE international symposium on empirical software engineering and measurement (ESE...

    2020

  47. [55]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2023 arXiv

  48. [56]

    OpenAI . 2024. o1 system card. https://cdn.openai.com/o1-system-card.pdf. Accessed: Dec 9, 2024

    2024

  49. [57]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 3...

    2022

  50. [58]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

    2023 arXiv

  51. [59]

    Kaiping Peng, Richard E Nisbett, and Nancy YC Wong. 1997. Validity problems comparing values across cultures and possible solutions. Psychological methods, 2(4):329

    1997

  52. [60]

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 2024. O1 replication journey: A strategic progress report--part 1. arXiv preprint arXiv:2410.18982

    2024 arXiv

  53. [61]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training

    2018

  54. [62]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  55. [63]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, p...

    2021

  56. [64]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p...

    2020

  57. [65]

    Martin Riddell, Ansong Ni, and Arman Cohan. 2024. Quantifying contamination in evaluating code generation capabilities of language models. arXiv preprint arXiv:2403.04811

    2024 arXiv

  58. [66]

    Oscar Sainz, Jon Ander Campos, Iker Garc \' a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. arXiv preprint arXiv:2310.18018

    2023 arXiv

  59. [67]

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146

    2024 arXiv

  60. [68]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  61. [69]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  62. [70]

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08115

    2024 arXiv

  63. [71]

    Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18--20, 2019, Proceedings 18, pages 194--206. Springer

    2019

  64. [72]

    Maciej \'S wiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Ma \'n dziuk. 2023. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):2497--2562

    2023

  65. [73]

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624

    2024 arXiv

  66. [74]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  67. [75]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  68. [76]

    O'Reilly Media, Inc

    Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. 2022. Natural language processing with transformers. " O'Reilly Media, Inc."

    2022

  69. [77]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, H Francis Song, Noah Yamamoto Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-based and outcome-based feedback

    2022

  70. [78]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  71. [79]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations

    2018

  72. [80]

    Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. 2024 a . Litesearch: Efficacious tree search for llm. arXiv preprint arXiv:2407.00320

    2024 arXiv

  73. [81]

    Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023 a . Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521

    2023 arXiv

  74. [82]

    Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. 2024 b . Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671

    2024 arXiv

  75. [83]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023 b . Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926

    2023 arXiv

  76. [84]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024 c . Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  77. [85]

    Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2024 d . Exploring vision-language models for imbalanced learning. International Journal of Computer Vision, 132(1):224--237

    2024

  78. [86]

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023 c . Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087

    2023 arXiv

  79. [87]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  80. [88]

    Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. 2024 e . Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658

    2024 arXiv

  81. [89]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  82. [90]

    Rui Xie, Zhengran Zeng, Zhuohao Yu, Chang Gao, Shikun Zhang, and Wei Ye. 2024. Codeshell technical report. arXiv preprint arXiv:2403.15747

    2024 arXiv

  83. [91]

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Watch every step! llm agent learning via iterative step-level process refinement. arXiv preprint arXiv:2406.11176

    2024 arXiv

  84. [92]

    Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2022. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073

    2022 arXiv

  85. [93]

    Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, et al. 2023. Supervised knowledge makes large language models better in-context learners. arXiv preprint arXiv:2312.15918

    2023 arXiv

  86. [94]

    Wenjin Yao, Yidong Wang, Zhuohao Yu, Rui Xie, Shikun Zhang, and Wei Ye. 2024. Pure: Aligning llm via pluggable query reformulation for enhanced helpfulness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8721--8744

    2024

  87. [95]

    Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. 2024 a . Kieval: A knowledge-grounded interactive evaluation framework for large language models. arXiv preprint arXiv:2402.15043

    2024 arXiv

  88. [96]

    Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, and Shikun Zhang. 2024 b . Freeeval: A modular framework for trustworthy and efficient evaluation of large language models. arXiv preprint arXiv:2404.06003

    2024 arXiv

  89. [97]

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816

    2024 arXiv

  90. [98]

    Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. arXiv preprint arXiv:2305.04087

    2023 arXiv

  91. [99]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  92. [100]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  93. [101]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906

    2024 arXiv

  94. [102]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  95. [103]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.