Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reward-guided search falls short because outcome reward models score partial and complete responses on different scales; this paper proves that a process reward model satisfying score consistency and preference consistency can recover the…

desk verdict Useful dual-consistency recipe for distilling PRMs from ORMs, but the preference-consistency filter leans on an unvalidated partial-sequence signal. read the letter →

arxiv 2506.12446 v2 pith:DABNTJGS submitted 2025-06-14 cs.CL

Bin Xie , Bingbing Xu , Yige Yuan , Shengmao Zhu , Huawei Shen This is my paper
classification cs.CL
keywords processrewardmodeloutcomescoreconsistencypreferencereward-guidedsearchinference-timealignmentpartialsequenceevaluationLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reward-guided search (RGS) fails to align language models because outcome reward models (ORMs) score complete responses but are used to score partial sequences during decoding, a granularity mismatch that produces myopic token choices. It claims an ideal process reward model (PRM) should satisfy two objectives: score consistency, meaning a preferred complete response has preferred prefixes at every length, and preference consistency, meaning partial-sequence scores should track human preferences. To get such a PRM without human annotation, it proposes SP-PRM, which re-trains an ORM on truncated preference pairs with a Bradley-Terry loss and reweights those pairs by an entropy-based confidence derived from a reference ORM. The paper reports that plugging this PRM into token-, chunk-, sentence-, and response-level RGS methods improves GPT-4 evaluation scores by 3.6–10.3% across dialogue, summarization, and reasoning tasks, and proves that a score-consistent reward model lets RGS recover the optimal response regardless of generation granularity.

What carries the argument

The load-bearing objects are the score-consistency condition (a preferred complete response must have preferred prefixes at every length) and the preference-consistency weighting derived from a reference ORM. SP-PRM operationalizes these with a Bradley-Terry loss over truncated partial pairs (with token-level truncation TLT or stochastic sampling truncation SST for efficiency), and an entropy-based sample weight $w_t = 1/H_t$ computed from the reference ORM's reward gap, which downweights short, low-confidence prefixes and discards samples where the reference disagrees with the score-consistency ordering. The proof of Theorem 1 uses induction over generation steps to show that a score-consistent reward model makes the argmax over next-token or chunk candidates coincide with the optimal response's next segment. The Agreement Rate metric $AR_{RM-SC}(t)$ measures how often a reward model's prefix ranking matches the score-consistency requirement, and is used to show the trained PRM improves over the original ORM.

What would settle it

On a held-out preference dataset where humans rank partial sequences (e.g., prefixes of length 10, 20, 50, and 100), compute the agreement between the reference ORM $r_{\phi}$ and human rankings; if agreement falls to around 50% (chance) for any prefix length in a domain the method is applied to, the preference-consistency weighting in Equation (3) would be driven by noise, and SP-PRM should fail to beat the plain ORM in reward-guided search on that domain.

Watch

Extended reading notes

Core claim

The central discovery is that the failure of outcome reward models in inference-time search is not just noise but a systematic granularity mismatch, and that a process reward model with two properties fixes it. Score consistency is formalized as: for any two sequences of equal length, if $r(x, y_1) \ge r(x, y_2)$ then $r(x, y_1^{<t}) \ge r(x, y_2^{<t})$ for every prefix length $t$; Theorem 1 shows that a reward model with this property can guide beam-style search to the globally optimal response at any granularity. Preference consistency is the additional requirement that partial-sequence scores agree with human preference judgments, which the paper shows existing ORMs satisfy only partially and score-consistent training alone can destroy. SP-PRM trains $r_{\theta}$ on truncated preference pairs with a Bradley-Terry loss, filters out pairs where a reference ORM $r_{\phi}$ disagrees with the score-consistency ordering, and weights the rest by $1/H_t$, the inverse Shannon entropy of the reward gap, so that more confident reference evaluations matter more. The result is a PRM that inherits the ORM's human-preference correlation while gaining score consistency, and the experiments show this improves token-, chunk-, sentence-, and response-level RGS methods across dialogue, summarization, and reasoning benchmarks.

Load-bearing premise

The method assumes that an existing outcome reward model, used as the reference $r_{\phi}$, provides trustworthy human-preference signals for partial sequences; if that transfer fails in another domain or for longer prefixes, the trained process reward model inherits the reference model's partial-sequence biases and the reported gains would not generalize.

Editorial extensions

If this is right

  • RGS methods that swap in SP-PRM's reward model should improve output quality without human preference annotation, because the PRM inherits the reference ORM's preference correlation while fixing the granularity mismatch.
  • All four granularities—token (ARGS), chunk (CBS), sentence (CARDS), and response (Best-of-N)—benefit from the same trained PRM, suggesting the fix is granularity-agnostic.
  • Because the preference-consistency module prevents the score-consistency objective from degrading semantic understanding, the method maintains diversity and coherence while raising reward scores; ablations that drop the reference-model guidance show 8–10% diversity drops.
  • Across 1B to 8B models in the paper's experiments, the trained reward models show higher agreement with score consistency at prefix lengths from 5 to 50 tokens, which the paper interprets as addressing myopic decoding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: on a new domain (e.g., code generation or tool-use), one could measure whether the reference ORM's partial-sequence agreement with human preferences stays above chance; if it does, SP-PRM should transfer, and if not, the entropy weighting would be weighting noise.
  • The paper's score-consistency condition compares prefixes of equal length only; extending it to variable-length prefixes or to stochastic or coarse-grained rewards could generalize Theorem 1 beyond the current equal-length padding convention.
  • The ceiling of SP-PRM is set by the reference ORM's own human-preference correlation; replacing $r_\phi$ with a stronger judge, or with human labels for a subset of prefixes, should directly raise the quality of the trained PRM, an implication of the authors' Observation 2 that they do not test.
  • The dual-consistency recipe is not limited to text: the same pattern of distilling a fine-grained, score-consistent critic from a coarse-grained reference evaluator could apply to any sequential decision problem where a trajectory-level evaluator is used to guide step-level choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes SP-PRM, a no-annotation training procedure that converts an outcome reward model (ORM) into a process reward model for inference-time reward-guided search (RGS). The method constructs partial-sequence pairs, trains a Bradley-Terry reward model with a score-consistency objective, and adds a preference-consistency filter that uses a reference ORM's partial-sequence ranking to discard and reweight samples. The authors prove that under an idealized score-consistency condition, an RGS procedure can recover the reward model's optimal response at any generation granularity, and they report experiments on HH-RLHF, AdvBench, TL;DR, and GSM8K showing improvements over ARGS, TBS/CBS, CARDS, and Best-of-N. The paper also includes ablations of the truncation strategy and the reference-model filter.

Significance. The paper identifies a real and practically important problem: ORMs are used to score partial sequences during guided decoding even though they were trained on complete responses. The proposed two-objective decomposition (score consistency and preference consistency) is intuitive, and Theorem 1 provides a clean sufficient condition for guided search to recover an optimal response. The experimental scope is broad, covering dialogue, safety, summarization, and reasoning with multiple RGS baselines and model sizes, and the code is released. The strengths are offset, however, by an evaluation metric that compares different reward functions across conditions, and by a reference-model filter whose partial-sequence reliability is validated only on HH-RLHF. As a result, the paper's central empirical claims are currently conditional.

major comments (4)
  1. [§5.2, Appendix C.2, Tables 1–4] The 'Average Reward' metric is explicitly computed with the reward model used during decoding. For the vanilla baselines this is the original ORM, while for '+Ours' conditions it is the SP-PRM-trained rθ, so the reward scores in the tables are not measured under a common evaluator. This makes the reported reward gains (e.g., the '15% to 25%' improvement described in §5.2.1) difficult to interpret and at least partly tautological. A concrete illustration is Table 1, BoN-16 for Llama-3.2-3B: the baseline Reward is 3.03 while '+Ours' is 2.89, yet the GPT-4 win-tie rate also decreases from 69.00 to 67.00; the two metrics point in opposite directions. I recommend reporting GPT-4 win-tie rates as the primary metric, and additionally scoring all methods with a fixed, held-out response-level reward model so that reward comparisons share the same evaluator.
  2. [§4.2, Eq. (3), with §3.2] The preference-consistency filter uses the reference ORM rφ's partial-sequence ranking as the ground truth for whether a partial pair is retained in the training set. This is exactly the granularity-mismatched signal the paper argues is unreliable: Observations 1 and 3 show that ORMs lack score consistency and that score-consistency orderings agree with human partial preferences less than 45% of the time at short prefixes. The only direct evidence for rφ's partial-prefix reliability is ARRM-HP >65% on HH-RLHF (Fig. 2a), and no such validation is reported for TL;DR, AdvBench, or GSM8K, which use different reference RMs (DeBERTa-v3-large, Gemma-2B, and Llama-based RMs). Since §5.3.2 shows that removing the reference-model filter substantially degrades results, the reported 3.6–10.3% gains depend on an untested transfer assumption. I request per-task prefix-level agreement rates (ARRM-HP, and ideally ARSC-HP) for each reference model, and an additional ablation that replaces Eq. (3) with a partial-preference signal that is not derived from the same ORM.
  3. [§3.1, Theorem 1, and Fig. 5] The theorem is stated for a reward model that satisfies score consistency for all pairs of sequences and all prefix lengths, which is a very strong condition. Fig. 5 shows that after SP-PRM training the agreement rate ARRM-SC is only about 55–65%, far from the 100% required by the theorem. Thus the theoretical result cannot be used to explain the empirical gains of the trained model; the paper should either explicitly label Theorem 1 as an idealized sufficient condition and provide a separate analysis of partial score consistency, or restrict the empirical claim to the subset of cases where the consistency condition is satisfied. Without this, the 'score consistency enables optimal generation' narrative is not connected to the actual trained reward models.
  4. [Tables 1–5 and §5.2] Most comparisons report no variance or significance tests. Diversity and Coherence are single numbers without error bars, and Win-tie, ROUGE-L, ASR, and Accuracy in Tables 3–5 are reported without confidence intervals. Some of the central differences are small (e.g., Table 1, CARDS '+Ours' vs '+Ablation' Win-tie 66.33 vs 66.67 for Llama-3.2-3B, and several GSM8K accuracy changes of 2–3.5 points in Table 4). Without multiple seeds or significance testing, the claim of 'substantial' improvement across all tasks is not fully supported. I recommend reporting standard deviations or confidence intervals for the main metrics and stating the number of evaluation seeds.
minor comments (6)
  1. [Abstract] The phrase 'GPT-4 evaluation scores across all tasks' is too broad; TL;DR results are reported via ROUGE-L and Average Reward, and AdvBench via ASR, not GPT-4 scores. Please specify which tables use GPT-4.
  2. [§4.1.1] The description of TLT says it either demands substantial computational resources or restricts sampling to under 5% of the original data; since TLT generates all prefixes, the 'sampling' language is unclear. Also for SST, sampling k times from U(1,T) can produce duplicate prefix lengths; state whether duplicates are removed.
  3. [§1 and §4] There are typographical errors ('their their', 'We proposeSP-PRM'), and 'DeBerta-v3-large' in §5.2.2 should be 'DeBERTa-v3-large'.
  4. [Appendix A, Theorem 1] The chunk-level proof assumes T is an integer multiple of L, but the main text pads sequences to equal length without specifying the padding token or how partial prefixes interact with padding; state this assumption explicitly in Theorem 1.
  5. [Figures 2 and 5] Axis labels and the exact prefix lengths are not described in the captions; please add them so the agreement-rate curves are interpretable.
  6. [Section 8 (Limitations)] The Limitations section lists only model-scale and inference-speed limitations; the unvalidated transfer of the preference-consistency filter and the reward-metric comparability issue should be acknowledged there.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity in the derivation chain; the only self-referential element is the 'Average Reward' metric, which uses the decoding reward model, but external metrics independently support the main improvement claim.

  1. other [Appendix C.2, Evaluation Metrics (see also Tables 1-4 Reward columns)]
    "Average Reward measures the mean RM scores across all test generations, calculated using the response-level reward models employed during decoding."

    The Reward metric is not independent of the method being evaluated: for the '+Ours' rows, the reward model used during decoding is the SP-PRM-trained r_theta (or its ORM teacher r_phi), and Eq. 3-4 explicitly optimize r_theta to agree with r_phi's partial-sequence preferences. Higher Average Reward can therefore reflect self-consistency between the trained PRM and the selection signal rather than external response quality. This makes the Reward columns partially circular as evidence. It is not derivation-collapsing, because the paper's headline gains are supported by external judgments (GPT-4 win-tie, ASR, ROUGE-L, Accuracy).

full rationale

The paper's core contribution is a training procedure (Eq. 2-4) that constructs partial-sequence pairs from preference data and trains a PRM via a Bradley-Terry objective, with a hard filter based on the reference ORM's partial-sequence ranking. This is a legitimate pipeline: the PRM is trained rather than defined in terms of the evaluation metric, and Theorem 1 is a genuine mathematical statement with an inductive proof that does not assume its conclusion. The self-citation in Section 6 merely lists prior work and is not load-bearing. The only circularity-adjacent issue is the Average Reward metric, which uses the same reward model family during decoding and training; however, the paper's main empirical claims are corroborated by external metrics (GPT-4 win-rate, ASR, ROUGE-L, Accuracy) that do not depend on the trained reward model, so the metric is a partial confound rather than a definitional collapse. Overall, no circular step is required to derive the paper's central result; the score reflects the minor evaluative self-reference.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. It relies on standard reward modeling assumptions, an empirical claim about ORM-human agreement, and a strong inductive bias that partial sequences inherit preference labels.

free parameters (2)
  • SST truncation hyperparameter K = 5
    Controls the number of truncated pairs per original preference pair (Section 4.1.1). Chosen by hand, not tuned on a validation set.
  • Reference reward model choice = Gemma-7B for HH-RLHF, DeBERTa-v3-large for TL;DR, etc.
    The reference model r_phi is a design choice that determines the preference filtering and entropy weights; different choices could change results.
assumptions (4)
  • standard math Bradley-Terry model as the reward modeling framework
    Used throughout for pairwise preference loss (Section 2.1).
  • domain assumption Existing ORMs maintain strong correlation with human preferences on partial sequences
    Observation 2 (Section 3.2) reports >65% agreement over prefix lengths for a few models, which underpins the use of r_phi for filtering and weighting.
  • domain assumption Truncated pairs inherit the preference label of the full response
    The training objective in Eq. 2 assumes y_w_<t is preferred over y_l_<t if y_w is preferred over y_l, which is exactly the score consistency assumption being imposed.
  • domain assumption Score consistency is sufficient to guarantee optimal reward-guided search
    Theorem 1 provides a proof under idealized conditions, but the trained model only approximates score consistency, so the theoretical guarantee does not directly apply.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment." pith.science (2026). https://pith.science/paper/DABNTJGS

@misc{pith2026250612446,
  author       = {Pith},
  title        = {Pith review of: From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DABNTJGS}},
  note         = {Machine review of arXiv:2506.12446}
}
read the original abstract

Inference-time alignment methods have gained significant attention for their efficiency and effectiveness in aligning large language models (LLMs) with human preferences. However, existing dominant approaches using reward-guided search (RGS) primarily rely on outcome reward models (ORMs), which suffer from a critical granularity mismatch: ORMs are designed to provide outcome rewards for complete responses, while RGS methods rely on process rewards to guide the policy, leading to inconsistent scoring and suboptimal alignment. To address this challenge, we introduce process reward models (PRMs) into RGS and argue that an ideal PRM should satisfy two objectives: Score Consistency, ensuring coherent evaluation across partial and complete responses, and Preference Consistency, aligning partial sequence assessments with human preferences. Based on these, we propose SP-PRM, a novel dual-consistency framework integrating score consistency-based and preference consistency-based partial evaluation modules without relying on human annotation. Extensive experiments on dialogue, summarization, and reasoning tasks demonstrate that SP-PRM substantially enhances existing RGS methods, achieving a 3.6%-10.3% improvement in GPT-4 evaluation scores across all tasks.

Figures

Figures reproduced from arXiv: 2506.12446 by the authors.

Figure 1
Figure 1. (Top) Inaccurate rewards of partial sequences [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Empirical Analysis of Reward Model Behaviors: (a) Existing ORMs maintain strong correlation with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the SP-PRM Framework. 2 Preliminaries In this section, we review the process of reward modeling and the general reward-guided search framework. 2.1 Reward Modeling We typically train a reward model on a preference dataset D. Each sample in D is represented as a triplet (x, yw, yl ), where x is the prompt, and y w and y l represent two distinct responses. Compared to y l , the response y w is more aligned… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of response generation between [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Score consistency (ARRM-SC) comparison be￾fore and after SP-PRM training across prefix lengths for different model (<1B to 3B). 5.4 Score Consistency Analysis We evaluate the score consistency of reward models before and after SP-PRM training [PITH_FULL_IMAGE:figures/…
Figure 6
Figure 6. Figure 6: GPT-4 Evaluation Template on HH-RLHF Dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: GPT-4 Evaluation Template on GSM8K Dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: An Example of Generated Texts using Different RGS Methods [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    GRPB ranks same-task video edits, converts the ranking into zero-sum advantages, and spreads them over editing segments through a lagged, capped allocator, producing a stronger editing agent.

Reference graph

Works this paper leans on

39 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  4. [4]

    Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. 2024. Transfer q star: Principled decoding for llm alignment. arXiv preprint arXiv:2405.20495

    arXiv 2024

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [6]

    Haikang Deng and Colin Raffel. 2023. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11781--11791

    work page 2023

  7. [7]

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236--1270

    work page 2023

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 39 references
  1. [9]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    2024 arXiv

  2. [10]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356--3369

    2020

  3. [11]

    Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. 2024. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405.06639

    2024 arXiv

  4. [12]

    James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. 2024. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06147

    2024

  5. [13]

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. 2024. Aligner: Efficient alignment by learning to correct. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  6. [14]

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. 2024. ARGS : Alignment as reward-guided search. In The Twelfth International Conference on Learning Representations

    2024

  7. [15]

    Bolian Li, Yifan Wang, Ananth Grama, and Ruqi Zhang. 2024. Cascade reward sampling for efficient decoding-time alignment. In ICML 2024 Next Generation of AI Safety Workshop

    2024

  8. [16]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  9. [17]

    Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02676

    2023 arXiv

  10. [18]

    Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. 2024 b . Inference-time language model alignment via integrated value guidance. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4181--4195

    2024

  11. [19]

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. 2023. Controlled decoding from language models. arXiv preprint arXiv:2310.17022

    2023 arXiv

  12. [20]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    2021 arXiv

  13. [21]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  14. [22]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

    2020

  16. [24]

    Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35:21548--21561

    2022

  17. [25]

    Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. 2024. Fast best-of-n decoding via speculative rejection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  18. [26]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  19. [27]

    Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, and Xipeng Qiu. 2024. I nfer A ligner: Inference-time alignment for harmlessness through cross-model guidance. Miami, Florida, USA. Association for Computational Linguistics

    2024

  20. [28]

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966

    2023 arXiv

  21. [29]

    Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. 2025 a . https://openreview.net/forum?id=jfwe9qNqRi Sim PER : A minimalist approach to preference alignment without hyperparameters . In The Thirteenth International Conferenc...

    2025

  22. [30]

    Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G Honavar. 2025 b . https://openreview.net/forum?id=2QdsjiNXgj On a connection between imitation learning and RLHF . In The Thirteenth International Conference on Learning Representations

    2025

  23. [31]

    Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. 2024. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. arXiv preprint arXiv:2410.08193

    2024 arXiv

  24. [32]

    Inference-time alignment in continuous space

    Yige Yuan, Teng Xiao, Li Yunfan, Xu Bingbing, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment

    2025

  25. [33]

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. 2023. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964

    2023 arXiv

  26. [34]

    Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. 2023. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708

    2023 arXiv

  27. [35]

    Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. 2024. Weak-to-strong search: Align large language models via searching over small language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  28. [36]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

  29. [37]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.