Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that post-training foundation models should be rebuilt as a search–verify–feedback loop driven by automated verifiers rather than primarily by human annotations.

desk verdict A useful survey taxonomy that reframes post-training as search, verify, and feedback, but the AGI claim leans on an unproven PAC assertion and the paper's own admission of weak verifier-performance correlation. read the letter →

arxiv 2411.11504 v1 pith:XH2KGETP submitted 2024-11-18 cs.AI cs.CLstat.ML

Xinyan Guan , Yanjiang Liu , Xinyu Lu , Boxi Cao , Ben He , Xianpei Han , Le Sun , Jie Lou
show 3 more authors
Bowen Yu Yaojie Lu Hongyu Lin
This is my paper · ORCID
classification cs.AIcs.CLstat.ML
keywords verifierengineeringpost-trainingfoundationmodelslargelanguagesearch-verify-feedbackloopgoal-conditionedMarkovdecisionprocessrewardreinforcementlearningfromhumanfeedback
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes verifier engineering as the next post-training paradigm for foundation models, after feature engineering and data engineering. The central idea is to replace most human annotation with a suite of automated verifiers—rule checkers, code interpreters, reward models, and judges—that evaluate candidate outputs and feed the results back to the model. The paper organizes the field into three stages: search, which samples candidate responses; verify, which scores or checks them; and feedback, which updates the model through training or inference-time guidance. It formalizes the loop as a goal-conditioned Markov decision process in which search, verify, and feedback correspond to action selection, reward evaluation, and policy optimization. The authors argue that this closed loop is the route to general intelligence.

What carries the argument

The load-bearing object is the goal-conditioned Markov decision process tuple $(S, A, T, G, R_g, p_g)$, with the reward decomposable as $R_g(s,a)=F(R_{g,i}(s,a)\mid i \in S_g)$, where each $R_{g,i}$ is a verifier for one capability dimension and $F$ combines them. Search corresponds to action selection in this MDP, verify to evaluation by the reward and verifier functions, and feedback to policy optimization. The paper's taxonomy of verifiers—by verification form (binary, score, rank, text), by granularity (token, thought, trajectory), by source (program-based or model-based), and by the need for extra training—delineates the design space from which verifier combinations are drawn. The combination step is justified by the paper's appeal to PAC theory, stated in Section 2.1, that ensembles of weak verifiers yield reliable overall evaluation.

What would settle it

A concrete check: assemble a suite of independent weak verifiers, adversarially search for responses that the majority rates correct but human experts judge wrong, and test whether such responses are systematically accepted by the combined verifier; if they are, the reliability assumption fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the next scaling step for foundation models is not more data or more parameters but a new source of supervision: automated verifiers. The paper names this paradigm verifier engineering and defines it as a closed loop of search, verify, and feedback: search generates diverse candidate responses, verify scores or checks them with a combination of verifiers, and feedback turns those results into either parameter updates (supervised fine-tuning, preference learning, reinforcement learning) or inference-time guidance (verifier-guided search, self-correction). It formalizes the loop as a goal-conditioned Markov decision process in which the reward function is decomposed over goal-specific verifier sub-functions, and it grounds the claim that even imperfect sub-functions can be combined into a reliable overall evaluation by appealing to PAC theory. The paper further claims that existing methods—RLHF, DPO, SFT, tree-of-thought-style search, and self-refinement—are all instances of this loop, and that verifier engineering is a fundamental pathway toward general intelligence.

Load-bearing premise

The load-bearing premise is that combining several imperfect verifiers—learned reward models, rule checkers, and code interpreters—yields a reliable evaluation signal, as the paper asserts by appealing to PAC theory.

Editorial extensions

If this is right

  • Human annotation can shrink from dense labeling to sparse verification, because a suite of automated verifiers supplies the supervision for both training and inference-time updates.
  • Methods as different as SFT, DPO, RLHF, step-wise verifiers, and tree search become instances of a single search–verify–feedback loop, so improvements in one stage could transfer across all of them.
  • The binding constraint on further model improvement shifts from data collection to verifier design, routing, and combination, making systematic verifier evaluation a critical benchmark problem.
  • Closed-loop improvement becomes possible without human labels: models can generate candidates, verify them with code interpreters or rule checkers, and update or refine from those signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: a direct empirical test of the paradigm would be to build a full search–verify–feedback loop on a single capability, such as mathematical word problems, using only open-source verifiers, and compare its closed-loop gain against the same budget spent on human-annotated supervised data.
  • Editorial: the framework implies a systems-level research agenda—automatic routing of instructions to verifier combinations and conflict resolution among their verdicts—that the paper describes but does not implement.
  • Editorial: if verifier ensembles are reliable only when their errors are decorrelated, the paradigm inherits an implicit diversity requirement; measuring error correlation across verifiers would sharpen the design space.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes "verifier engineering" as a post-training paradigm for foundation models, in which a suite of automated verifiers replaces or supplements human annotation as the main source of supervision. The paradigm is formalized as a goal-conditioned MDP whose reward function is decomposed into sub-functions, and the authors organize the field into three stages—Search, Verify, and Feedback—with a large table (Table 3) mapping many existing methods to these stages. The paper also revisits SFT, DPO, and RLHF from this perspective and concludes that verifier engineering is a fundamental pathway toward AGI.

Significance. If the framework is accepted, it offers a useful unifying vocabulary for post-training research and highlights a concrete design space (verifier construction, combination, and feedback) beyond the current reward-model-centric view of RLHF. The survey is broad, with extensive coverage of recent work, and the GC-MDP formalization provides a common notation for comparing methods. The paper ships a GitHub repository collecting the categorized references, which is a concrete resource. However, the central theoretical justification for the Verify stage rests on an unproven PAC-based claim, and the paper itself acknowledges evidence that verifier quality does not reliably predict downstream gains. These gaps directly affect the strength of the paradigm claim, making the paper more a position/taxonomy contribution than a demonstrated technical result.

major comments (3)
  1. [§2.1, Eq. (2)] The sentence "Following Probably Approximately Correct Theory (PAC), even with imperfect sub-functions, we can achieve reliable overall evaluation by combining multiple weak verifiers" is stated without derivation or a specific theorem reference. PAC theory gives generalization bounds for a learned hypothesis class under i.i.d. assumptions; it does not, by itself, imply that an arbitrary combination of learned reward models, rule checkers, and tools yields a reliable reward function R_g. This claim is load-bearing for the Verify stage and for the AGI claim. Please either provide a concrete theorem with conditions (e.g., independence assumptions, error-rate bounds, or a calibrated aggregation rule) under which F is reliable, or explicitly rephrase this as a heuristic design principle rather than a theoretical guarantee.
  2. [§6.2.2] The paper acknowledges that "recent studies (Wen et al., 2024c) have revealed only weak correlations between existing verifier evaluation metrics and downstream task performance." This admission directly undercuts the assumption that the Verify stage supplies supervision that reliably improves the policy in the Search-Verify-Feedback loop. The paper does not provide a theoretical bridge from verifier accuracy to policy improvement, nor experimental evidence that the loop outperforms standard RLHF. To make the central claim defensible, the authors should either present such evidence or explicitly scope the claim to settings where verifier-policy alignment has been established, and discuss concrete failure modes such as reward hacking and over-optimization (which are only mentioned in passing in §6.2.3).
  3. [Abstract and §7] The claim that verifier engineering "constitutes a fundamental pathway toward achieving Artificial General Intelligence" goes beyond what the survey supports. The paper provides a taxonomy and qualitative examples, but no argument that verifier engineering is necessary or sufficient for general intelligence, nor a definition of AGI against which the claim could be evaluated. This overclaim weakens the paper's credibility. I suggest reframing the statement as "a promising and scalable paradigm for post-training foundation models" and, if the AGI connection is to be retained, supporting it with a substantive argument about the scalability of supervision and its relation to general capability.
minor comments (4)
  1. [§1, paragraph 4] The sentence "The remarkable success of landmark projects such as ImageNet (Deng et al., 2009) and BERT (Brown et al., 2020a)" misattributes BERT; Brown et al. 2020 is the GPT-3 paper. The correct citation for BERT is Devlin et al., 2018.
  2. [Table 3] There are several typographical inconsistencies: "STar" should be "STaR", "Constrative decoding" should be "Contrastive decoding", and the use of both "Self-Debug" and "Self-debug" across rows should be unified.
  3. [Figure 3 caption] The caption says gray nodes represent sample paths not used in training, but it is not immediately clear from the figure which nodes are gray in each of (a), (b), and (c); please increase the contrast or label the paths explicitly.
  4. [§2.1] The text states that "in large language models, the state transition is a deterministic function" but then describes search as "exploration in the action-state space." This is not contradictory, but it would be clearer to note that determinism is with respect to the transition given a chosen action, while exploration arises from the policy's stochastic action selection.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is a position/taxonomy paper whose GC-MDP mapping is a definitional organizational translation rather than a derived prediction, and whose self-citations are illustrative rather than load-bearing.

full rationale

This is a position/survey paper rather than a derivation-and-prediction paper. The GC-MDP formalization in Section 2 defines the reward R_g as the verification result and then maps Search, Verify, and Feedback onto action selection, reward, and policy optimization; that mapping is an organizational translation, not an empirical prediction, so it does not manufacture a result by construction. The PAC-based assertion in Section 2.1 ('Following Probably Approximately Correct Theory (PAC), even with imperfect sub-functions, we can achieve reliable overall evaluation by combining multiple weak verifiers') is genuinely under-derived, and the paper itself weakens it in Section 6.2.2 by noting that 'recent studies (Wen et al., 2024c) have revealed only weak correlations between existing verifier evaluation metrics and downstream task performance'; however, an unsupported assumption or a validity gap is not circularity, because the paper does not define verifier reliability in terms of its AGI conclusion. Self-citations (Cao et al., Guan et al., Lu et al., Wen et al., and others) appear as taxonomy examples or as a cited limitation, and none of them carries the burden of the central framework claim. No step reduces by construction to its own inputs, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces the concept label 'verifier engineering' but no new physical or formal entities such as particles, forces, or dimensions. The axioms listed are the load-bearing background assumptions that the taxonomy and the AGI claim depend on.

assumptions (4)
  • domain assumption Combining multiple weak verifiers yields a reliable overall evaluation signal.
    Invoked in Section 2.1 after Eq. 2 with a citation to PAC theory, but no proof is given that arbitrary combinations of imperfect verifiers concentrate on the true quality.
  • domain assumption State transitions in LLM generation can be modeled as deterministic functions.
    Section 2.1 states 'the state transition is a deterministic function.' This holds for greedy decoding but not for stochastic sampling, which is common in search.
  • ad hoc to paper All relevant post-training methods decompose cleanly into the three stages Search, Verify, Feedback.
    The framework is defined so broadly that nearly every cited method is categorized in Table 3; this is an assumption of the taxonomy rather than a proven property.
  • domain assumption Automated verifiers can scale as supervision signals without the same cost as human annotation.
    The paper's motivation in the Introduction, but no evidence shows that verifier feedback avoids reward hacking or remains aligned as the policy changes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering." pith.science (2026). https://pith.science/paper/XH2KGETP

@misc{pith2026241111504,
  author       = {Pith},
  title        = {Pith review of: Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XH2KGETP}},
  note         = {Machine review of arXiv:2411.11504}
}
read the original abstract

The evolution of machine learning has increasingly prioritized the development of powerful models and more scalable supervision signals. However, the emergence of foundation models presents significant challenges in providing effective supervision signals necessary for further enhancing their capabilities. Consequently, there is an urgent need to explore novel supervision signals and technical approaches. In this paper, we propose verifier engineering, a novel post-training paradigm specifically designed for the era of foundation models. The core of verifier engineering involves leveraging a suite of automated verifiers to perform verification tasks and deliver meaningful feedback to foundation models. We systematically categorize the verifier engineering process into three essential stages: search, verify, and feedback, and provide a comprehensive review of state-of-the-art research developments within each stage. We believe that verifier engineering constitutes a fundamental pathway toward achieving Artificial General Intelligence.

Figures

Figures reproduced from arXiv: 2411.11504 by the authors.

Figure 1
Figure 1. Framework of verifier engineering: The fundamental stages of verifier engineering include Search, Verify, and Feedback. Given an instruction, the process begins with generating candidate responses (Search), followed by evaluating these candidates using appropriate verifier combinations (Verify), and concludes with optimizing the model’s output distribution (Feedback). This framework can explain various approaches, f… view at source ↗
Figure 2
Figure 2. Overview of verifier engineering methodologies, categorized into three main stages: Search, Verify, and Feedback. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A verifier engineering perspective on SFT, DPO, [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoning

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A checkpoint-based search and candidate augmentation method improves small LLM mathematical reasoning accuracy over existing test-time scaling baselines.

  2. Beyond Theorem Proving: Formulation, Framework and Benchmark for Formal Problem-Solving

    cs.AI 2025-05 conditional novelty 7.0 of 10

    Formulates problem-solving as a sound Markov decision process, implements it in Lean as FPS and D-FPS, and introduces three formal problem-solving benchmarks plus the RPE answer-equivalence checker.

  3. IRPO: Boosting Image Restoration via Post-training GRPO

    cs.CV 2025-11 conditional novelty 6.0 of 10

    GRPO post-training on the worst 30% of samples with a mixed fidelity/perceptual reward improves AdaIR by 0.83 dB in-domain and 3.43 dB on out-of-domain benchmarks.

  4. DeepRAG: Thinking to Retrieve Step by Step for Large Language Models

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A retrieval-augmented QA system that learns when to retrieve at each reasoning step outperforms adaptive RAG baselines on six benchmarks.

  5. Reasoning Language Models: A Blueprint

    cs.AI 2025-01 accept novelty 5.0 of 10

    A modular blueprint and open-source framework (x1) that presents existing reasoning language model designs as special cases of one unified toolbox.

Reference graph

Works this paper leans on

149 extracted references · 5 canonical work pages · cited by 5 Pith papers

  1. [1]

    Measuring the persuasiveness of language models, 2024

    Anthropic. Measuring the persuasiveness of language models, 2024. https://www.anthropic.com/research/measuring-model-persuasiveness/

    2024

  2. [2]

    A general language assistant as a laboratory for alignment

    Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. ArXiv preprint, abs/2112.00861, 2021. URL https://arxiv.org/abs/2112.00861

    arXiv 2021

  3. [3]

    G., Guo, Z

    Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    2024

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.05862, 2022 a . URL https://arxiv.org/abs/2204.05862

    arXiv 2022

  5. [5]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. ArXiv preprint, abs/2212.08073, 2022 b . URL https://arxiv.org/abs/2212.08073

    arXiv 2022

  6. [6]

    Improving image generation with better captions

    Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., Jiao, Y., and Ramesh, A. Improving image generation with better captions. URL https://api.semanticscholar.org/CorpusID:264403242

  7. [7]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324, 1952. URL https://api.semanticscholar.org/CorpusID:125209808

    1952

  8. [8]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert - Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford...

    2020

Show all 149 references
  1. [9]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., L...

    2020 arXiv

  2. [10]

    H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J

    Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023. URL https://arxiv.org/abs/2312.09390

    2023 arXiv

  3. [11]

    Internlm2 technical report

    Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

    2024 arXiv

  4. [12]

    Towards scalable automated alignment of llms: A survey, 2024

    Cao, B., Lu, K., Lu, X., Chen, J., Ren, M., Xiang, H., Liu, P., Lu, Y., He, B., Han, X., Sun, L., Lin, H., and Yu, B. Towards scalable automated alignment of llms: A survey, 2024. URL https://arxiv.org/abs/2406.01252

    2024 arXiv

  5. [13]

    Code alpaca: An instruction-following llama model for code generation

    Chaudhary, S. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    2023

  6. [14]

    A., Chan, J

    Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., and Perez, E. Improving code generation by training with natural language feedback, 2024 a . URL https://arxiv.org/abs/2303.16749

    2024 arXiv

  7. [15]

    Alphamath almost zero: process supervision without process

    Chen, G., Liao, M., Li, C., and Fan, K. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553, 2024 b

    2024 arXiv

  8. [16]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2023 a . URL https://arxiv.org/abs/2211.12588

    2023 arXiv

  9. [17]

    Teaching large language models to self-debug, 2023 b

    Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug, 2023 b

    2023

  10. [18]

    Teaching large language models to self-debug, 2023 c

    Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug, 2023 c . URL https://arxiv.org/abs/2304.05128

    2023 arXiv

  11. [19]

    X., Wan, J., Zhang, F., Zhang, D., and Wen, J.-R

    Chen, Z., Zhou, K., Zhao, W. X., Wan, J., Zhang, F., Zhang, D., and Wen, J.-R. Improving large language models via fine-grained reinforcement learning with minimum editing constraint, 2024 c

    2024

  12. [20]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  13. [21]

    Training verifiers to solve math word problems, 2021

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021

    2021

  14. [22]

    Efficient selectivity and backup operators in monte-carlo tree search

    Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp.\ 72--83. Springer, 2006

    2006

  15. [23]

    M., Kong, S., Avigad, J., van Doorn, F., and von Raumer, J

    de Moura, L. M., Kong, S., Avigad, J., van Doorn, F., and von Raumer, J. The lean theorem prover (system description). In CADE, 2015. URL https://api.semanticscholar.org/CorpusID:232990

    2015

  16. [24]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  17. [25]

    Chain-of-verification reduces hallucination in large language models, 2023

    Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. Chain-of-verification reduces hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.11495

    2023 arXiv

  18. [26]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023

    2023 arXiv

  19. [27]

    RAFT : Reward ranked finetuning for generative foundation model alignment

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. RAFT : Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/for...

    2023

  20. [28]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/ab...

    2021 arXiv

  21. [29]

    Mods: Model-oriented data selection for instruction tuning

    Du, Q., Zong, C., and Zhang, J. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653, 2023

    2023 arXiv

  22. [30]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  23. [31]

    Kto: Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  24. [32]

    Hierarchical neural story generation, 2018

    Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation, 2018. URL https://arxiv.org/abs/1805.04833

    2018 arXiv

  25. [33]

    and Krell, M

    Friedland, G. and Krell, M. A capacity scaling law for artificial neural networks, 2018. URL https://arxiv.org/abs/1708.06019

    2018 arXiv

  26. [34]

    Pal: Program-aided language models

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. In International Conference on Machine Learning, pp.\ 10764--10799. PMLR, 2023

    2023

  27. [36]

    Critic: Large language models can self-correct with tool-interactive critiquing, 2024

    Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., and Chen, W. Critic: Large language models can self-correct with tool-interactive critiquing, 2024. URL https://arxiv.org/abs/2305.11738

    2024 arXiv

  28. [37]

    Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting, 2023

    Guan, X., Liu, Y., Lin, H., Lu, Y., He, B., Han, X., and Sun, L. Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting, 2023. URL https://arxiv.org/abs/2311.13314

    2023 arXiv

  29. [38]

    Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting

    Guan, X., Liu, Y., Lin, H., Lu, Y., He, B., Han, X., and Sun, L. Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18126--18134, 2024

    2024

  30. [39]

    A., Dumais, S

    Hearst, M. A., Dumais, S. T., Osuna, E., Platt, J., and Scholkopf, B. Support vector machines. IEEE Intelligent Systems and their applications, 13 0 (4): 0 18--28, 1998

    1998

  31. [40]

    Distilling the knowledge in a neural network

    Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  32. [41]

    The curious case of neural text degeneration, 2020

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration, 2020. URL https://arxiv.org/abs/1904.09751

    2020 arXiv

  33. [42]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  34. [43]

    T., He, J., Koh, P

    Hu, Z., Liu, C., Feng, X., Zhao, Y., Ng, S.-K., Luu, A. T., He, J., Koh, P. W., and Hooi, B. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models, 2024. URL https://arxiv.org/abs/2402.03271

    2024 arXiv

  35. [44]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  36. [45]

    Jiang, D., Ren, X., and Lin, B. Y. LLM -blender: Ensembling large language models with pairwise ranking and generative fusion. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volum...

    2023 doi

  37. [46]

    F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G

    Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023 b

    2023 arXiv

  38. [47]

    Jordan, M. I. and Mitchell, T. M. Machine learning: Trends, perspectives, and prospects. Science, 349 0 (6245): 0 255--260, 2015

    2015

  39. [48]

    and Szepesv \'a ri, C

    Kocsis, L. and Szepesv \'a ri, C. Bandit based monte-carlo planning. In European conference on machine learning, pp.\ 282--293. Springer, 2006

    2006

  40. [49]

    Reformulating unsupervised style transfer as paraphrase generation

    Krishna, K., Wieting, J., and Iyyer, M. Reformulating unsupervised style transfer as paraphrase generation. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 737--762, Onlin...

    2020 doi

  41. [50]

    D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., Zhang, L

    Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., Zhang, L. M., McKinney, K., Shrivastava, D., Paduraru, C., Tucker, G., Precup, D., Behbahani, F., and Faust, A. Training language models to self-correct via ...

    2024 arXiv

  42. [51]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  43. [52]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023

    Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023

    2023

  44. [53]

    Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L

    Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp.\ 3197-...

    2022

  45. [55]

    Scalable agent alignment via reward modeling: a research direction, 2018 b

    Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward modeling: a research direction, 2018 b . URL https://arxiv.org/abs/1811.07871

    2018 arXiv

  46. [56]

    Fast inference from transformers via speculative decoding, 2023

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding, 2023. URL https://arxiv.org/abs/2211.17192

    2023 arXiv

  47. [58]

    L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M

    Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Contrastive decoding: Open-ended text generation as optimization, 2023 a . URL https://arxiv.org/abs/2210.15097

    2023 arXiv

  48. [59]

    Making language models better reasoners with step-aware verifier

    Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., and Chen, W. Making language models better reasoners with step-aware verifier. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  49. [60]

    RAIN : Your language models can align themselves without finetuning

    Li, Y., Wei, F., Zhao, J., Zhang, C., and Zhang, H. RAIN : Your language models can align themselves without finetuning. In The Twelfth International Conference on Learning Representations, volume abs/2309.07124, 2024 a . URL https://openreview.net/forum?id=pETSfWMUzy

    2024 arXiv

  50. [61]

    Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structurization, 2024 b

    Li, Z., Chen, X., Yu, H., Lin, H., Lu, Y., Tang, Q., Huang, F., Han, X., Sun, L., and Li, Y. Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structurization, 2024 b . URL https://arxiv.org/abs/2410.08815

    2024 arXiv

  51. [62]

    Mario: Math reasoning with code interpreter output -- a reproducible pipeline, 2024

    Liao, M., Luo, W., Li, C., Wu, J., and Fan, K. Mario: Math reasoning with code interpreter output -- a reproducible pipeline, 2024

    2024

  52. [63]

    Let's verify step by step

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023 a

    2023 arXiv

  53. [64]

    Let's verify step by step, 2023 b

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step, 2023 b

    2023

  54. [65]

    Lean-star: Learning to interleave thinking and proving, 2024

    Lin, H., Sun, Z., Yang, Y., and Welleck, S. Lean-star: Learning to interleave thinking and proving, 2024

    2024

  55. [67]

    Rltf: Reinforcement learning from unit test feedback, 2023 b

    Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., and Ye, D. Rltf: Reinforcement learning from unit test feedback, 2023 b . URL https://arxiv.org/abs/2307.04349

    2023 arXiv

  56. [68]

    Goal-conditioned reinforcement learning: Problems and solutions, 2022

    Liu, M., Zhu, M., and Zhang, W. Goal-conditioned reinforcement learning: Problems and solutions, 2022. URL https://arxiv.org/abs/2201.08299

    2022 arXiv

  57. [69]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=BTKAeLqLMw

    2024

  58. [70]

    Quark: Controllable text generation with reinforced unlearning, 2022

    Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning, 2022

    2022

  59. [71]

    Transferable post-training via inverse value learning, 2024

    Lu, X., Wen, X., Lu, Y., Yu, B., Lin, H., Yu, H., Sun, L., Han, X., and Li, Y. Transferable post-training via inverse value learning, 2024. URL https://arxiv.org/abs/2410.21027

    2024 arXiv

  60. [72]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023 a

    Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023 a

    2023

  61. [74]

    Improve mathematical reasoning in language models by automated process supervision

    Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024 b

    2024 arXiv

  62. [75]

    Python is not always the best choice: Embracing multilingual program of thoughts, 2024 c

    Luo, X., Zhu, Q., Zhang, Z., Qin, L., Zhang, X., Yang, Q., Xu, D., and Che, W. Python is not always the best choice: Embracing multilingual program of thoughts, 2024 c . URL https://arxiv.org/abs/2402.10691

    2024 arXiv

  63. [76]

    Wizardcoder: Empowering code large language models with evol-instruct

    Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023 b

    2023 arXiv

  64. [77]

    Wizardcoder: Empowering code large language models with evol-instruct

    Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024 d . URL https://openreview.net/forum?i...

    2024

  65. [78]

    P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh...

    2023

  66. [79]

    Self-refine: Iterative refinement with self-feedback

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  67. [80]

    E., Lee, T., Adler, S., Jiang, A., and Weng, L

    Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T., Adler, S., Jiang, A., and Weng, L. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 15009--15018, 2023

    2023

  68. [81]

    and Xin, M

    Mo, S. and Xin, M. Tree of uncertain thoughts reasoning for large language models, 2023. URL https://arxiv.org/abs/2309.07694

    2023 arXiv

  69. [82]

    Rule based rewards for language model safety

    Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I., Lin, M., Beutel, A., Schulman, J., and Weng, L. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 2024

    2024

  70. [83]

    C., Liu, Z., Xue, L., Yao, W., Feng, Y., Chen, Z., Gokul, A., Arpit, D., Xu, R., Mui, P., Wang, H., Xiong, C., and Savarese, S

    Murthy, R., Heinecke, S., Niebles, J. C., Liu, Z., Xue, L., Yao, W., Feng, Y., Chen, Z., Gokul, A., Arpit, D., Xu, R., Mui, P., Wang, H., Xiong, C., and Savarese, S. Rex: Rapid exploration and exploitation for ai agents, 2024. URL https://arxiv.org/abs/2307.08962

    2024 arXiv

  71. [84]

    Webgpt: Browser-assisted question-answering with human feedback

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  72. [85]

    and Satoh, K

    Nguyen, H.-T. and Satoh, K. Balancing exploration and exploitation in llm using soft rllf for enhanced negation understanding. arXiv preprint arXiv:2403.01185, 2024

    2024 arXiv

  73. [86]

    and Lewis, M

    O'Brien, S. and Lewis, M. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117, 2023

    2023 arXiv

  74. [87]

    Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers

    Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., and Levy, R. Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proceedings of the 2023 Conference on Empirical Methods in Natural La...

    2023 doi

  75. [88]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022 a

    2022

  76. [89]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to f...

    2022 arXiv

  77. [90]

    Not all contexts are equal: Teaching llms credibility-aware generation, 2024

    Pan, R., Cao, B., Lin, H., Han, X., Zheng, J., Wang, S., Cai, X., and Sun, L. Not all contexts are equal: Teaching llms credibility-aware generation, 2024. URL https://arxiv.org/abs/2404.06809

    2024 arXiv

  78. [91]

    Language model self-improvement by reinforcement learning contemplation, 2023

    Pang, J.-C., Wang, P., Li, K., Chen, X.-H., Xu, J., Zhang, Z., and Yu, Y. Language model self-improvement by reinforcement learning contemplation, 2023. URL https://arxiv.org/abs/2305.14483

    2023 arXiv

  79. [92]

    Instruction tuning with gpt-4, 2023

    Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4, 2023

    2023

  80. [93]

    Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018

    Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., Kumar, V., and Zaremba, W. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018. URL https://arxiv.or...

    2018 arXiv

  81. [94]

    Experiential co-learning of software-developing agents

    Qian, C., Dang, Y., Li, J., Liu, W., Xie, Z., Wang, Y., Chen, W., Yang, C., Cong, X., Che, X., et al. Experiential co-learning of software-developing agents. arXiv preprint arXiv:2312.17025, 2023

    2023 arXiv

  82. [95]

    Dmoerm: Recipes of mixture-of-experts for effective reward modeling

    Quan, S. Dmoerm: Recipes of mixture-of-experts for effective reward modeling. arXiv preprint arXiv:2403.01197, 2024

    2024 arXiv

  83. [96]

    Quinlan, J. R. Induction of decision trees. Machine learning, 1: 0 81--106, 1986

    1986

  84. [97]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  85. [98]

    The probabilistic relevance framework: Bm25 and beyond

    Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3 0 (4): 0 333--389, 2009

    2009

  86. [99]

    Self-critiquing models for assisting human evaluators

    Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators. ArXiv preprint, abs/2206.05802, 2022. URL https://arxiv.org/abs/2206.05802

    2022 arXiv

  87. [100]

    Universal value function approximators

    Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp.\ 1312--1320, Lille, Fran...

    2015

  88. [101]

    Deep learning in neural networks: An overview

    Schmidhuber, J. Deep learning in neural networks: An overview. Neural Networks, 61: 0 85--117, 2015. ISSN 0893-6080. doi:https://doi.org/10.1016/j.neunet.2014.09.003. URL https://www.sciencedirect.com/science/article/pii/S0893608014002135

    2015 doi

  89. [102]

    Proximal policy optimization algorithms, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017

    2017

  90. [103]

    Compute trends across three eras of machine learning

    Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., and Villalobos, P. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pp.\ 1–8. IEEE, July 2022. doi:10.1109/ijcnn55064.2022.9891914. URL http://dx...

    2022

  91. [105]

    Shojaee, P., Jain, A., Tipirneni, S., and Reddy, C. K. Execution-based code generation using deep reinforcement learning, 2023 b . URL https://arxiv.org/abs/2301.13816

    2023 arXiv

  92. [106]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  93. [107]

    Preference ranking optimization for human alignment

    Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18990--18998, 2024

    2024

  94. [108]

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  95. [109]

    Fast best-of-n decoding via speculative rejection, 2024 a

    Sun, H., Haider, M., Zhang, R., Yang, H., Qiu, J., Yin, M., Wang, M., Bartlett, P., and Zanette, A. Fast best-of-n decoding via speculative rejection, 2024 a . URL https://arxiv.org/abs/2410.20290

    2024 arXiv

  96. [110]

    The critique of critique

    Sun, S., Li, J., Yuan, W., Yuan, R., Li, W., and Liu, P. The critique of critique. ArXiv preprint, abs/2401.04518, 2024 b . URL https://arxiv.org/abs/2401.04518

    2024 arXiv

  97. [111]

    Z., Zheng, Z., Calandriello, D., Cao, Y., Tarassov, E., Munos, R., Pires, B

    Tang, Y., Guo, D. Z., Zheng, Z., Calandriello, D., Cao, Y., Tarassov, E., Munos, R., Pires, B. \'A ., Valko, M., Cheng, Y., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448, 2024

    2024 arXiv

  98. [112]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  99. [113]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  100. [114]

    F., Zhou, X., Liu, S

    Tsai, C. F., Zhou, X., Liu, S. S., Li, J., Yu, M., and Mei, H. Can large language models play text games well? current state-of-the-art and open questions. arXiv preprint arXiv:2304.02868, 2023

    2023 arXiv

  101. [115]

    Vapnik, V. N. The Nature of Statistical Learning Theory. Springer, 2000

    2000

  102. [116]

    K., Cogswell, M., Selvaraju, R

    Vijayakumar, A. K., Cogswell, M., Selvaraju, R. R., Sun, Q., Lee, S., Crandall, D., and Batra, D. Diverse beam search: Decoding diverse solutions from neural sequence models, 2018. URL https://arxiv.org/abs/1610.02424

    2018 arXiv

  103. [118]

    Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning, 2023 b

    Wang, K., Ren, H., Zhou, A., Lu, Z., Luo, S., Shi, W., Zhang, R., Song, L., Zhan, M., and Li, H. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning, 2023 b . URL https://arxiv.org/abs/2310.03731

    2023 arXiv

  104. [120]

    X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z

    Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024 a

    2024

  105. [121]

    X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z

    Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024 b . URL https://arxiv.org/abs/2312.08935

    2024 arXiv

  106. [122]

    and Zhou, D

    Wang, X. and Zhou, D. Chain-of-thought reasoning without prompting. ArXiv preprint, abs/2402.10200, 2024. URL https://arxiv.org/abs/2402.10200

    2024 arXiv

  107. [123]

    V., Chi, E

    Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023 d . URL https://openreview.net/...

    2023

  108. [124]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association ...

    2023 doi

  109. [125]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  110. [127]

    R., He, H., and Feng, S

    Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, S. R., He, H., and Feng, S. Language models learn to mislead humans via rlhf, 2024 b . URL https://arxiv.org/abs/2409.12822

    2024 arXiv

  111. [128]

    Rethinking reward model evaluation: Are we barking up the wrong tree? arXiv preprint arXiv:2410.05584, 2024 c

    Wen, X., Lou, J., Lu, Y., Lin, H., Yu, X., Lu, X., He, B., Han, X., Zhang, D., and Sun, L. Rethinking reward model evaluation: Are we barking up the wrong tree? arXiv preprint arXiv:2410.05584, 2024 c

    2024 arXiv

  112. [129]

    On-policy fine-grained knowledge feedback for hallucination mitigation, 2024 d

    Wen, X., Lu, X., Guan, X., Lu, Y., Lin, H., He, B., Han, X., and Sun, L. On-policy fine-grained knowledge feedback for hallucination mitigation, 2024 d . URL https://arxiv.org/abs/2406.12221

    2024 arXiv

  113. [130]

    Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences, 2024

    Weyssow, M., Kamanda, A., and Sahraoui, H. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences, 2024

    2024

  114. [131]

    Llm tree search

    Wilson, D. Llm tree search. arXiv preprint arXiv:2410.19117, 2024

    2024 arXiv

  115. [132]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge

    Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024

    2024 arXiv

  116. [133]

    Aligning large language models via self-steering optimization, 2024

    Xiang, H., Yu, B., Lin, H., Lu, K., Lu, Y., Han, X., Sun, L., Zhou, J., and Lin, J. Aligning large language models via self-steering optimization, 2024. URL https://arxiv.org/abs/2410.17131

    2024 arXiv

  117. [134]

    Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling

    Xie, T., Ma, Y., and Wang, Y.-X. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems, 32, 2019

    2019

  118. [135]

    P., Kawaguchi, K., and Shieh, M

    Xie, Y., Goyal, A., Zheng, W., Kan, M.-Y., Lillicrap, T. P., Kawaguchi, K., and Shieh, M. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 2024

    2024 arXiv

  119. [136]

    Beyond full fine-tuning: Harnessing the power of L o RA for multi-task instruction tuning

    Xin, C., Lu, Y., Lin, H., Zhou, S., Zhu, H., Wang, W., Liu, Z., Han, X., and Sun, L. Beyond full fine-tuning: Harnessing the power of L o RA for multi-task instruction tuning. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the...

    2024

  120. [137]

    Baize: An open-source chat model with parameter-efficient tuning on self-chat data

    Xu, C., Guo, D., Duan, N., and McAuley, J. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196, 2023

    2023 arXiv

  121. [138]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022

    2022

  122. [139]

    Tree of thoughts: Deliberate problem solving with large language models

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  123. [140]

    OVM , outcome-supervised value models for planning in mathematical reasoning

    Yu, F., Gao, A., and Wang, B. OVM , outcome-supervised value models for planning in mathematical reasoning. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp.\ 858--875, Mexico City, Mexico, June 2024. Asso...

    2024 doi

  124. [141]

    T., Li, Z., Weller, A., and Liu, W

    Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023 a

    2023 arXiv

  125. [142]

    Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation

    Yu, Z., Zhang, X., Shang, N., Huang, Y., Xu, C., Zhao, Y., Hu, W., and Yin, Q. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint arXiv:2312.14187, 2023 b

    2023 arXiv

  126. [143]

    RRHF : Rank responses to align language models with human feedback

    Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. RRHF : Rank responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.net/forum?id=EdIGMCHk4l

    2023

  127. [144]

    Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. ArXiv preprint, abs/2401.10020, 2024. URL https://arxiv.org/abs/2401.10020

    2024 arXiv

  128. [145]

    Scaling relationship on learning mathematical reasoning with large language models, 2023 b

    Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathematical reasoning with large language models, 2023 b

    2023

  129. [146]

    Scaling relationship on learning mathematical reasoning with large language models, 2023 c

    Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathematical reasoning with large language models, 2023 c . URL https://arxiv.org/abs/2308.01825

    2023 arXiv

  130. [147]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  131. [148]

    Star: Bootstrapping reasoning with reasoning

    Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Bootstrapping reasoning with reasoning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 15476--15488. Curran Associates, In...

    2022

  132. [149]

    Star: Bootstrapping reasoning with reasoning

    Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 0 15476--15488, 2022 b

    2022

  133. [150]

    Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., and Goodman, N. D. Quiet-star: Language models can teach themselves to think before speaking. ArXiv preprint, abs/2403.09629, 2024. URL https://arxiv.org/abs/2403.09629

    2024 arXiv

  134. [151]

    and Church, R

    Zeng, W. and Church, R. L. Finding shortest paths on real road networks: the case for a*. Int. J. Geogr. Inf. Sci., 23 0 (4): 0 531–543, April 2009. ISSN 1365-8816. doi:10.1080/13658810801949850. URL https://doi.org/10.1080/13658810801949850

    2009 doi

  135. [152]

    Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning, 2024 a

    Zhang, D., Wu, J., Lei, J., Che, T., Li, J., Xie, T., Huang, X., Zhang, S., Pavone, M., Li, Y., Ouyang, W., and Zhou, D. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning, 2024 a . URL https://arxiv.org/abs/2410.02884

    2024 arXiv

  136. [153]

    Rest-mcts*: Llm self-training via process reward guided tree search, 2024 b

    Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, J. Rest-mcts*: Llm self-training via process reward guided tree search, 2024 b . URL https://arxiv.org/abs/2406.03816

    2024 arXiv

  137. [155]

    Generative verifiers: Reward modeling as next-token prediction, 2024 d

    Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative verifiers: Reward modeling as next-token prediction, 2024 d . URL https://arxiv.org/abs/2408.15240

    2024 arXiv

  138. [156]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2024 a . URL https://proceedings.neurip...

    2024

  139. [157]

    Secrets of rlhf in large language models part i: Ppo, 2023

    Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., Xiong, L., Chen, L., Xi, Z., Xu, N., Lai, W., Zhu, M., Chang, C., Yin, Z., Weng, R., Cheng, W., Huang, H., Sun, T., Yan, H., Gui, T., Zhang, Q., Qiu, X., and Huang, X. Secrets of rlh...

    2023

  140. [158]

    Y., Fu, J., Chen, W., and Yue, X

    Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter: Integrating code generation with execution and refinement, 2024 b

    2024

  141. [159]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.