Pith. sign in

REVIEW 8 cited by

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.08193 v5 pith:RCDZCAHE submitted 2024-10-10 cs.CL

Yuancheng Xu , Udari Madhushani Sehwag , Alec Koppel , Sicheng Zhu , Bang An , Furong Huang , Sumitra Ganesh This is my paper
classification cs.CL
keywords alignmentgenarmllmsrewardtest-timeautoregressivegenerationmethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) exhibit impressive capabilities but require careful alignment with human preferences. Traditional training-time methods finetune LLMs using human preference datasets but incur significant training costs and require repeated training to handle diverse user preferences. Test-time alignment methods address this by using reward models (RMs) to guide frozen LLMs without retraining. However, existing test-time approaches rely on trajectory-level RMs which are designed to evaluate complete responses, making them unsuitable for autoregressive text generation that requires computing next-token rewards from partial responses. To address this, we introduce GenARM, a test-time alignment approach that leverages the Autoregressive Reward Model--a novel reward parametrization designed to predict next-token rewards for efficient and effective autoregressive generation. Theoretically, we demonstrate that this parametrization can provably guide frozen LLMs toward any distribution achievable by traditional RMs within the KL-regularized reinforcement learning framework. Experimental results show that GenARM significantly outperforms prior test-time alignment baselines and matches the performance of training-time methods. Additionally, GenARM enables efficient weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high costs of training larger models. Furthermore, GenARM supports multi-objective alignment, allowing real-time trade-offs between preference dimensions and catering to diverse user preferences without retraining. Our project page is available at: https://genarm.github.io.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    REFORM uses reward-guided controlled decoding to generate preference-class-consistent responses that the reward model mis-scores, then retrains the reward model on these failure modes to improve robustness.

  2. Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Sampling from a sharpened, vision-calibrated target during decoding improves frozen LVLM reasoning accuracy by 2-3 points on five benchmarks, but multiplies token cost by 15-24.

  3. Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

    cs.LG 2025-10 conditional novelty 6.0 of 10

    MAHALO aligns LLMs to multiple objectives in one model via per-objective action heads and PRM-guided decoding, improving math, value, and tutoring metrics jointly.

  4. From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SP-PRM trains process reward models from outcome reward models via score and preference consistency, improving reward-guided search on dialogue, summarization, and reasoning tasks.

  5. Inference-Time Policy Alignment for Fair Reinforcement Learning

    cs.LG 2026-07 reject novelty 5.0 of 10

    A frozen RL policy can be reweighted at test time by a learned generalized-Gini welfare critic to improve fairness metrics, though the central equivalence mixes up two different welfare objectives.

  6. Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective

    cs.LG 2026-01 conditional novelty 5.0 of 10

    The optimal reward for KL-regularized LLM alignment is a threshold function—reward B above a prompt-dependent cutoff, 0 below—which can be estimated from base-model samples and integrated into decoding-time alignment.

  7. A Survey on Training-free Alignment of Large Language Models

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.

  8. UCD: Unlearning in LLMs via Contrastive Decoding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.

Pith tools