Pith. sign in

REVIEW 10 cited by

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.08193 v5 pith:RCDZCAHE submitted 2024-10-10 cs.CL

Yuancheng Xu , Udari Madhushani Sehwag , Alec Koppel , Sicheng Zhu , Bang An , Furong Huang , Sumitra Ganesh This is my paper · ORCID
classification cs.CL
keywords alignmentgenarmllmsrewardtest-timeautoregressivegenerationmethods
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) exhibit impressive capabilities but require careful alignment with human preferences. Traditional training-time methods finetune LLMs using human preference datasets but incur significant training costs and require repeated training to handle diverse user preferences. Test-time alignment methods address this by using reward models (RMs) to guide frozen LLMs without retraining. However, existing test-time approaches rely on trajectory-level RMs which are designed to evaluate complete responses, making them unsuitable for autoregressive text generation that requires computing next-token rewards from partial responses. To address this, we introduce GenARM, a test-time alignment approach that leverages the Autoregressive Reward Model--a novel reward parametrization designed to predict next-token rewards for efficient and effective autoregressive generation. Theoretically, we demonstrate that this parametrization can provably guide frozen LLMs toward any distribution achievable by traditional RMs within the KL-regularized reinforcement learning framework. Experimental results show that GenARM significantly outperforms prior test-time alignment baselines and matches the performance of training-time methods. Additionally, GenARM enables efficient weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high costs of training larger models. Furthermore, GenARM supports multi-objective alignment, allowing real-time trade-offs between preference dimensions and catering to diverse user preferences without retraining. Our project page is available at: https://genarm.github.io.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    REFORM uses reward-guided controlled decoding to generate preference-class-consistent responses that the reward model mis-scores, then retrains the reward model on these failure modes to improve robustness.

  2. Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Sampling from a sharpened, vision-calibrated target during decoding improves frozen LVLM reasoning accuracy by 2-3 points on five benchmarks, but multiplies token cost by 15-24.

  3. Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

    cs.LG 2025-10 conditional novelty 6.0 of 10

    MAHALO aligns LLMs to multiple objectives in one model via per-objective action heads and PRM-guided decoding, improving math, value, and tutoring metrics jointly.

  4. Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach

    cs.LG 2025-06 conditional novelty 6.0 of 10

    IRO trains a sequence of small value functions that reweight a frozen LLM's own samples, achieving test-time alignment gains that outperform Best-of-N search on AlpacaEval 2.0 and TL;DR.

  5. From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SP-PRM trains process reward models from outcome reward models via score and preference consistency, improving reward-guided search on dialogue, summarization, and reasoning tasks.

  6. Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A TD-trained vision value model guides sentence-level inference-time search in VLMs, cutting hallucination and improving caption quality, with self-training gains on nine benchmarks.

  7. Inference-Time Policy Alignment for Fair Reinforcement Learning

    cs.LG 2026-07 reject novelty 5.0 of 10

    A frozen RL policy can be reweighted at test time by a learned generalized-Gini welfare critic to improve fairness metrics, though the central equivalence mixes up two different welfare objectives.

  8. Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective

    cs.LG 2026-01 conditional novelty 5.0 of 10

    The optimal reward for KL-regularized LLM alignment is a threshold function—reward B above a prompt-dependent cutoff, 0 below—which can be estimated from base-model samples and integrated into decoding-time alignment.

  9. A Survey on Training-free Alignment of Large Language Models

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.

  10. UCD: Unlearning in LLMs via Contrastive Decoding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.

Pith tools