Inference-Time Policy Alignment for Fair Reinforcement Learning

Conor Wallace; Peilang Li; Umer Siddique; Yongcan Cao

REVIEW 3 major objections 3 minor 27 references

A pretrained RL policy can be made fairer at deployment by reweighting its action probabilities with a welfare critic, with no parameter updates to the base policy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:05 UTC pith:BX53GY7K

load-bearing objection Good problem, plausible method, but the central theorem equates E[φ(R)] with φ(E[R]), so the stated guarantee doesn't follow; the paper is repairable by redefining the objective. the 3 major comments →

arxiv 2608.00175 v1 pith:BX53GY7K submitted 2026-07-31 cs.LG cs.AI

Inference-Time Policy Alignment for Fair Reinforcement Learning

Umer Siddique , Peilang Li , Conor Wallace , Yongcan Cao This is my paper

classification cs.LG cs.AI

keywords inference-time alignmentpolicy shapingfair reinforcement learninggeneralized Gini welfare functionmulti-objective MDPwelfare criticKL-regularized policytest-time adaptation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that fairness—specifically, a Gini-style welfare measure over multiple reward objectives—can be imposed on an already-trained reinforcement learning policy at inference time, by reweighting the policy's action distribution with a learned critic while leaving the base policy's parameters untouched. The central claim is that this reweighting is not a heuristic: it is the closed-form solution to a KL-regularized policy improvement problem in a 'welfare-augmented' MDP whose state includes the cumulative reward vector. To make that tractable, the paper introduces QFair, an offline-trained critic that estimates welfare action-values conditioned on the augmented state. If correct, the result means any deep RL agent—value-based or policy-gradient—can be steered toward new fairness preferences without retraining, which matters for deployment settings where preferences emerge after training or differ across clients. The paper validates the claim in three environments, showing shaped policies reduce inequality across objectives while keeping total reward competitive.

Core claim

On its own terms, the paper's discovery is that the non-Markovian problem of maximizing the generalized Gini welfare of expected return can be converted into a Markovian one by augmenting the state with the accrued reward vector, and that the optimal KL-regularized policy in that augmented MDP takes the closed form π'(a|s,R) ∝ π(a|s) exp(Q̃^π((s,R),a)/β). The shaped policy is computed directly from the frozen base policy and a welfare critic; no iterative policy updates are needed. The paper further claims that, because base-policy rollouts can be annotated with marginal welfare rewards and reused offline, the critic can be trained without additional environment interaction under the welfare

What carries the argument

The load-bearing object is the welfare-augmented MDP: state (s,R) with R the cumulative reward vector, and reward defined as the marginal welfare contribution φ_w(R+r) − φ_w(R). Telescoping makes the sum of these marginal rewards equal φ_w(R_T) − φ_w(0), linking the augmented-MDP return to the welfare objective. The second piece is Theorem 4.2's closed-form shaped policy, a Boltzmann reweighting of the base policy by the welfare Q-function: π'(a|s,R) ∝ π(a|s) exp((1/β) Q̃^π((s,R),a)). This exponential form is what the practical QFair critic implements via a softmax over base-policy logits plus λ times normalized welfare scores.

Load-bearing premise

The proof that maximizing the shaped policy's return equals maximizing the stated welfare objective assumes that the welfare score of the average outcome equals the average of the welfare scores—true for linear scores but not for the non-linear generalized Gini function used here, so the objective actually optimized at inference time may differ from the fairness metric reported.

What would settle it

Take any trained shaped policy from the paper's Species Conservation domain, roll it out for many seeds, and compute E[φ_w(R_T)] and φ_w(E[R_T]) from the terminal return vectors. If the two quantities differ by more than the evaluation noise—which they should whenever outcomes are unequal—then Theorem 4.1's equivalence claim, and hence the identification of the shaped policy's return with the stated fairness objective, is violated in the very setting the paper evaluates.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

A pretrained policy can serve multiple stakeholders with different fairness preferences by swapping in the critic's welfare weights at deployment, instead of training one policy per preference.
Fairness alignment no longer needs to be baked into training, so a single reward-maximizing base policy can be adaptively steered when regulatory or societal preferences change.
Because the shaped policy preserves the support of the base policy, the approach inherits the base policy's competence guarantees—actions outside the base support are never introduced.
The QFair critic can be trained entirely offline from base-policy rollouts, avoiding additional environment interaction under the welfare objective.
The framework applies to any deep RL algorithm—demonstrated with DQN, A2C, and PPO—as long as the base policy's action distribution can be queried.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the equivalence between the augmented-MDP return and the stated objective holds strictly only for the expected-welfare variant; for the paper's stated GGF-of-expected-return objective, the two differ by Jensen's inequality, so a sharper evaluation would report E[φ_w(R)] and φ_w(E[R]) separately.
Editorial inference: the same reweighting mechanism could transfer to other trajectory-level objectives (e.g., risk-sensitive or safety constraints) whenever a critic can be trained on cumulative-state-dependent scores, since the KL-regularized derivation does not depend on the specific form of φ_w.
Editorial inference: the softmax formulation suggests a natural extension to continuous action spaces via importance-weighted reweighting or normalizing flows; the current implementation requires enumerating actions.
Editorial inference: because the critic is trained on base-policy rollouts, distribution shift from base to shaped policy could produce biased welfare estimates; a testable extension is to iteratively re-collect data under the shaped policy and retrain the critic.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Good problem, plausible method, but the central theorem equates E[φ(R)] with φ(E[R]), so the stated guarantee doesn't follow; the paper is repairable by redefining the objective.

read the letter

You should know this paper before citing it: the problem is real and the method is plausible, but the main theorem has a Jensen gap that breaks the stated equivalence.

What's genuinely new: test-time multiplicative shaping of a frozen RL policy using a critic conditioned on the cumulative reward vector. That's not in the cited prior work, which either trains the GGF during learning or uses classifiers for short-horizon text agents. The augmented-MDP construction is clean, and the closed-form shaped policy in Theorem 4.2 is a standard KL-regularized policy update—that part is correct conditional on the surrogate.

The experiments are solid for a conference paper: three domains, three base algorithms (PPO, A2C, DQN), and GGF-trained baselines. The shaped variants often improve CV and welfare while keeping total reward competitive, and the hyperparameters are reported, though code is not released.

Now the soft spots, in proportion. The load-bearing flaw is Theorem 4.1. Section 3.2 defines the objective as Jω(π') = φω(E[R_T]) — welfare of expected return. The telescoping sum in Theorem 4.1 gives E~π[φω(R_T)] − φω(0). These are equal only if φ commutes with expectation. For nonlinear GGF, Jensen's inequality gives E[φ(R)] ≤ φ(E[R]) for concave φ, with strict inequality generically. The appendix proof stops at the telescoping identity and then asserts the equivalence. So Eq. (7)'s guarantee — that the shaped policy maximizes the stated welfare objective — does not follow. The paper itself says in Section 3.2 that expected-welfare optimization "cannot theoretically ensure fairness," which makes the slip more striking.

Second, QFair is trained with Expected SARSA under behavior policy μ = (1−β)π + β uniform, so it approximates Q^μ, not Q^π. The theorem requires Q^π. There's no error bound or correction for the mismatch. It might be a minor issue in practice, but it's a real gap between theory and implementation.

Third, the evaluation applies φ to the empirical mean return, which is consistent with the stated objective φ(E[R]) but not with the theorem's E[φ(R)]. So the numbers don't validate the theory either.

A mild note on circularity: the GGF defines both the critic's reward and the evaluation metric, so some welfare improvement is by construction. That's an optimization objective, not a derivation, so I don't weight it heavily.

The paper is repairable. Reformulating the objective as expected welfare E[φ(R_T)] would make Theorem 4.1 go through, at the cost of changing the evaluation. Alternatively, analyzing the welfare-of-expected-return objective directly is much harder and would likely need distributional or risk-sensitive tools. With the expected-welfare reformulation, the method becomes a plausible surrogate.

This paper is for researchers working on fairness in RL and test-time alignment. They get a clear formulation and a promising empirical recipe, but the central theoretical claim is not supported as written. I would send it to peer review — the problem is important, the method is plausible, and the flaw is identifiable and repairable, not a dead end. But I would expect major revision, not acceptance.

Referee Report

3 major / 3 minor

Summary. The paper proposes inference-time alignment of a frozen pretrained RL policy to a generalized-Gini welfare objective. The authors introduce a welfare-augmented MDP whose state includes the cumulative reward vector and whose reward is the marginal welfare increment. They claim (Thm 4.1) that maximizing return in this augmented MDP is equivalent to maximizing J_ω(π') = φ_ω(E[R_T]), and then derive a closed-form shaped policy (Thm 4.2) that reweights the base policy by the welfare Q-function. A learned critic (QFair) is trained offline with Expected SARSA under an exploratory behavior policy. Experiments on three environments compare shaped PPO/A2C/DQN with standard RL and GGF-trained baselines.

Significance. The problem is timely and the proposed pipeline is appealing: if the theory were correct, it would give a simple, general way to adapt arbitrary pretrained deep RL policies to fairness at deployment. The paper's strengths are its clear formalization of the inference-time alignment problem, the construction of a welfare-augmented MDP, the closed-form shaping rule, and the breadth of the empirical study across value-based and policy-gradient algorithms. However, the central equivalence is mathematically incorrect as stated, and the evaluation metric does not match the objective that the theory actually optimizes. The QFair training also targets a different policy than the one required by the theoretical shaping formula. These issues are load-bearing rather than cosmetic.

major comments (3)

[Sec. 3.2, Eq. (2); Thm 4.1, Eq. (3); Eq. (15)] The paper defines the fairness objective as J_ω(π') = φ_ω(E[R_T]) (welfare of the expected return). Theorem 4.1 proves that the total reward in fM telescopes to E[φ_ω(R_T)] - φ_ω(0). These two quantities are not equal for nonlinear GGF: for concave φ_ω, Jensen's inequality gives E[φ_ω(R_T)] ≤ φ_ω(E[R_T]), with strict inequality under return variance. The 'Hence' in Theorem 4.1 and the bridge identity Eq. (15) therefore do not follow. Appendix A.1's proof also stops at the telescoping identity and asserts the equivalence without an expectation/welfare commutation step. This invalidates the guarantee for Theorem 4.2 / Eq. (7).
[Sec. 6, Figs. 4-7; Sec. 3.2] Evaluation applies φ to the empirical mean return vector, i.e., φ(mean_t R_T), which is the welfare-of-expected-return objective. But the augmented-MDP theory optimizes E[φ(R_T)] (expected welfare). For nonlinear φ these differ, so the reported welfare improvements do not validate the stated objective. If the authors intend to repair the theory by adopting E[φ(R_T)] as the objective, the evaluation must instead average per-trajectory φ(R_T) values; with the current metric the empirical claims are mismatched with the theory.
[Sec. 5.1, Eq. (8); Thm 4.2, Eq. (7)] The optimal shaped policy in Eq. (7) uses Q̃^π, the welfare action-value function of the lifted base policy. QFair is trained with Expected SARSA under the behavior policy μ=(1-β)π+β uniform, so it converges to Q̃^μ, not Q̃^π. The statement that Q̃^μ is 'close to' Q̃^π is unsupported and no off-policy correction or error bound is provided. As a result, the implemented shaping is not the closed-form solution derived in Theorem 4.2.

minor comments (3)

[Notation] The welfare function is written φ_ω in Section 3.2 and Eq. (1), but φ_w in Definition 4.1, Eq. (3), and the appendix. Please use a single consistent symbol.
[Algorithm 2, Eq. (9)] The base-policy signal f_base(a) is left unspecified for value-based agents such as DQN. If it is log π(a|s), say so; if it is a Q-value, clarify how it maps onto the theoretical exponential reweighting of Eq. (7).
[Figures 4-5] The text reports results over 20 seeds in Species Conservation, but Figures 4 and 5 do not show error bars or confidence intervals, making it difficult to judge whether the apparent improvements in CV and welfare are statistically meaningful.

Circularity Check

0 steps flagged

No significant circularity: the derivation chain does not reduce to its inputs by construction.

full rationale

The paper's central derivation is not circular. Theorem 4.2 is obtained from a standard KL-regularized policy-improvement surrogate in the augmented MDP, and the closed-form shaped policy follows from the Lagrangian/KKT argument rather than being assumed as the conclusion. The welfare critic QFair is trained by TD on marginal-welfare rewards derived from the GGF, and its use in shaping is a learned optimization step, not a fitted parameter renamed as a prediction. The GGF itself is introduced from the external literature (Weymark 1981) and standard welfare axioms, and the self-citations (e.g., Siddique et al. 2020) are used for baselines, hyperparameters, and related work rather than as load-bearing evidence for the main claim. The most serious issue in the paper is a correctness gap, not circularity: Theorem 4.1's telescoping reward yields E[phi(R_T)], while the stated objective is phi(E[R_T]); the claimed equivalence would require E[phi(R_T)] = phi(E[R_T]), which is false for nonlinear GGF by Jensen's inequality. This is a non-sequitur or objective mismatch, but it is not an identity created by definition, so it does not constitute circularity under the analysis rules.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central method rests on two untested premises: the equivalence between expected welfare and welfare of expected return (false), and sufficient coverage of the augmented state space by lightly explored base rollouts. The free parameters (GGF weights, λ, exploration) are not fully reported.

free parameters (3)

GGF weight vector w = not reported in experiments
The welfare function (Eq 1) requires strictly decreasing positive weights; the paper does not state the values used, and the trade-off between equality and efficiency depends on them.
alignment strength λ (inverse temperature) = not reported
Eq (9) and Algorithm 2 use λ to blend base and fairness signals; Appendix D lists it as a hyperparameter but omits values across environments, so the reported results cannot be reproduced.
behavior-policy exploration β_explore = 0.1 for DQN; unreported for PPO/A2C
Offline data collection mixes the base policy with uniform exploration (Section 5.1); the exploration rate controls coverage of the augmented state space and is part of the method.

axioms (5)

domain assumption Vector reward r∈R^N is observable at deployment and the scalar training reward is its sum.
Assumption (ii) in Section 4; without this decomposition the QFair critic cannot be trained or applied.
domain assumption The frozen base policy π can be queried as a full action distribution.
Assumption (i); shaping requires the base probabilities to reweight them (Eq 7).
ad hoc to paper E[φ(R_T)] = φ(E[R_T]) for the GGF welfare function.
Implicit in the jump from Eq (3) to 'equivalent to maximizing J_ω' in Theorem 4.1; false for nonlinear φ. This is the paper's central unstated premise.
standard math The KL-regularized surrogate (Eq 5-6) is a valid local approximation to the true welfare objective.
Standard trust-region argument (Schulman et al., 2015), but only applies to the objective actually optimized.
domain assumption Behavior policy μ with light exploration provides sufficient coverage for TD learning in the augmented MDP.
Section 5.1 asserts light exploration is enough for TD generalization; no coverage or concentration guarantee is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 23626 in / 14531 out tokens · 154764 ms · 2026-08-04T01:05:04.661603+00:00 · methodology

0 comments

read the original abstract

Deep reinforcement learning (RL) agents achieve strong performance by optimizing scalar reward functions. However, once deployed, the policies of these RL agents are often rigid and costly to adapt to new performance criteria. For instance, an agent trained to maximize expected cumulative reward may not accommodate previously unknown stakeholder preferences. Existing approaches to achieve fairness, a type of preference, in RL typically assume that such preferences are known a priori and require complete retraining of the policy under a fairness-oriented metric. Inspired by inference-time alignment in large language models, we investigate the problem of steering a pretrained RL policy toward welfare-based fairness objectives at inference time without updating the base policy's parameters. We formalize inference-time fairness alignment as a policy shaping problem and propose a multiplicative policy shaping framework that adjusts action probabilities using action-dependent welfare scores, thus requiring no modification to the base policy. Our framework is general and compatible with any deep RL agent. Through extensive experiments across multiple domains, we demonstrate that inference-time policy shaping substantially improves welfare-based fairness objectives while preserving core task performance.

Figures

Figures reproduced from arXiv: 2608.00175 by Conor Wallace, Peilang Li, Umer Siddique, Yongcan Cao.

**Figure 2.** Figure 2: Example where different cumulative rewards lead to different optimal actions. Example 4.1 To illustrate Proposition 4.1, we consider an MDP with N = 2 objectives and GGF weights w1 > w2 > 0. As seen in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Inference-time policy shaping pipeline. At each step, the current state [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Performance comparison of PPO, A2C, and DQN with their inference-time shaped vari [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

**Figure 5.** Figure 5: Performance comparison of PPO, A2C, DQN, their inference-time shaped variants, and [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

**Figure 6.** Figure 6: Performance comparison of PPO, A2C, DQN, their inference-time shaped variants, and [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗

**Figure 7.** Figure 7: Performance comparison of PPO, A2C, DQN, their inference-time shaped variants, and [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

27 extracted references · 11 linked inside Pith

[1]

Summing (26) overa: 1 = exp c(˜s) X a π(a|s) exp 1 β eA˜π(˜s, a) ,(27) Reinforcement Learning Journal 2026 so: exp c(˜s) = 1 P a π(a|s) exp 1 β eA˜π(˜s, a) .(28) Substituting back into (26): ˜π′(a|˜s) = π(a|s) exp 1 β eA˜π(˜s, a) P a′ π(a′|s) exp 1 β eA˜π(˜s, a′) .(29) Now we replace the advantage eA˜πwith the Q-function eQ˜π. Since eA˜π(˜s, a) =eQ˜π(˜s, ...

2026
[7]

Out-of- distribution detection for reinforcement learning agents with probabilistic dynamics models

Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann. Out-of- distribution detection for reinforcement learning agents with probabilistic dynamics models. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Sys- tems, pp. 851–859,

2023
[9]

Llm post-training: A deep dive into reasoning large language models.arXiv preprint arXiv:2502.21321,

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models.arXiv preprint arXiv:2502.21321,

Pith/arXiv arXiv
[12]

Scalable multi-objective reinforcement learning with fairness guarantees using lorenz dominance

Dimitris Michailidis, Willem Röpke, Diederik M Roijers, Sennay Ghebreab, and Fernando P Santos. Scalable multi-objective reinforcement learning with fairness guarantees using lorenz dominance. arXiv preprint arXiv:2411.18195,

arXiv
[19]

Steering llm reasoning through bias-only adaptation

Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov, Boris Shaposhnikov, Nikita Balagansky, and Daniil Gavrilov. Steering llm reasoning through bias-only adaptation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 9213–9222,

2025
[21]

Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,

Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,

Pith/arXiv arXiv
[23]

A Theoretical Analysis of Inference-Time Fairness Alignment In this appendix, we provide the complete theoretical analysis of our inference-time fairness align- ment framework

Reinforcement Learning Journal 2026 Supplementary Materials The following content was not necessarily subject to peer review. A Theoretical Analysis of Inference-Time Fairness Alignment In this appendix, we provide the complete theoretical analysis of our inference-time fairness align- ment framework. For clarity and self-containment, we recall the core p...

2026
[24]

The lifting gives˜π(at |˜st) =π(a t |s t), and ePmarginalises toP(s ′ |s, a)since theR-update is deterministic

QT−1 t=0 π(at |s t)P(s t+1 |s t, at). The lifting gives˜π(at |˜st) =π(a t |s t), and ePmarginalises toP(s ′ |s, a)since theR-update is deterministic. A direct and powerful consequence of Proposition A.2 is that rollouts collected by the base policy can bedirectly reusedfor offline training in fM. We simply annotate each transition with the cumula- tive re...

2026
[27]

In practice,α >0and the constraint is active

deter- minesα: eitherα= 0(the unconstrained optimum already satisfiesKL≤ε) or the KL constraint binds (equality holds) andα >0. In practice,α >0and the constraint is active. Determiningβ.The parameterβ=α >0is implicitly defined by the constraint: E˜s∼d˜π KL ˜π′ β(·|˜s) ˜π(·|˜s) =ε,(31) where˜π′ β is the solution (30) parameterised byβ. The left-hand side ...

2026
[28]

To ensure robust performance and a fair baseline comparison, we tuned the hyperparam- eters for these agents using the Optuna optimization framework. For the multi-objective baselines that optimize the welfare function directly during training (i.e., GGF-PPO, GGF-A2C, and GGF- DQN), we adopted the author-provided hyperparameters detailed in (Siddique et a...

2020
[1998]

Do- main randomization for transferring deep neural networks from simulation to the real world

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23–30. IEEE,

2017
[2002]

applied to fM, we can express the welfare gap aseJ(˜π′)− eJ(˜π) = TE ˜s∼d˜π′ h eA˜π(˜s, a) i . Sinced ˜π′ depends on the unknown policy, we replace it withd ˜πto obtain a tractable welfare surrogate that shares the same gradient at˜π ′ = ˜πand provides a local lower bound on the true improvement under a KL penalty (Schulman et al., 2015). Therefore, to ma...

2015
[2004]

Aligning machiavellian agents: Behavior steering via test-time policy shaping.arXiv preprint arXiv:2511.11551,

Dena Mujtaba, Brian Hu, Anthony Hoogs, and Arslan Basharat. Aligning machiavellian agents: Behavior steering via test-time policy shaping.arXiv preprint arXiv:2511.11551,

arXiv
[2010]

Decaf: Learning to be fair in multi-agent resource allocation

Ashwin Kumar and William Yeoh. Decaf: Learning to be fair in multi-agent resource allocation. arXiv preprint arXiv:2502.04281,

Pith/arXiv arXiv
[2012]

Welfare and fairness in multi-objective reinforcement learning.arXiv preprint arXiv:2212.01382,

Zimeng Fan, Nianli Peng, Muhang Tian, and Brandon Fain. Welfare and fairness in multi-objective reinforcement learning.arXiv preprint arXiv:2212.01382,

Pith/arXiv arXiv
[2013]

A survey on personalized align- ment—the missing piece for large language models in real-world applications

Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey on personalized align- ment—the missing piece for large language models in real-world applications. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 5313–5333,

2025
[2015]

Policy shaping: Integrating human feedback with reinforcement learning.Advances in neural information processing systems, 26,

Reinforcement Learning Journal 2026 Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning.Advances in neural information processing systems, 26,

2026
[2016]

Genarm: Reward guided generation with autoregressive reward model for test-time alignment.arXiv preprint arXiv:2410.08193,

Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment.arXiv preprint arXiv:2410.08193,

Pith/arXiv arXiv
[2017]

URLhttp://arxiv.org/abs/ 1707.06347. Reinforcement Learning Journal 2026 Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

Pith/arXiv arXiv 2026
[2018]

Inference-time pol- icy adapters (ipa): Tailoring extreme-scale lms without fine-tuning

Policy Shaping for Fair RL Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, et al. Inference-time pol- icy adapters (ipa): Tailoring extreme-scale lms without fine-tuning. InProceedings of the 2023 conference on empirical methods in natural language ...

2023
[2020]

Fairness in preference-based reinforcement learning

Umer Siddique, Abhinav Sinha, and Yongcan Cao. Fairness in preference-based reinforcement learning. InICML 2023 Workshop The Many Facets of Preference-Based Learning,

2023
[2021]

Latent activation editing: Inference-time refinement of learned policies for safer multirobot navigation

Satyajeet Das, Darren Chiu, Zhehui Huang, Lars Lindemann, and Gaurav S Sukhatme. Latent activation editing: Inference-time refinement of learned policies for safer multirobot navigation. arXiv preprint arXiv:2509.20623,

Pith/arXiv arXiv
[2022]

Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,

Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hut- ter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,

Pith/arXiv arXiv
[2023]

Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model.arXiv preprint arXiv:2505.06274,

Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, and Ying-Cong Chen. Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model.arXiv preprint arXiv:2505.06274,

Pith/arXiv arXiv
[2024]

Towards fair and efficient policy learning in cooper- ative multi-agent reinforcement learning

Umer Siddique, Peilang Li, and Yongcan Cao. Towards fair and efficient policy learning in cooper- ative multi-agent reinforcement learning. InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 2744–2746, 2025a. Umer Siddique, Peilang Li, and Yongcan Cao. Learning fair pareto-optimal policies in multi- objecti...

Pith/arXiv arXiv
[2025]

Optimizing generalized gini indices for fairness in rankings.arXiv preprint arXiv:2204.06521,

Virginie Do and Nicolas Usunier. Optimizing generalized gini indices for fairness in rankings.arXiv preprint arXiv:2204.06521,

Pith/arXiv arXiv
[2026]

Shifting perspectives: Steer- ing vectors for robust bias mitigation in llms.arXiv preprint arXiv:2503.05371, 2025b

Zara Siddique, Irtaza Khalid, Liam D Turner, and Luis Espinosa-Anke. Shifting perspectives: Steer- ing vectors for robust bias mitigation in llms.arXiv preprint arXiv:2503.05371, 2025b. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Maste...

arXiv

[1] [1]

Summing (26) overa: 1 = exp c(˜s) X a π(a|s) exp 1 β eA˜π(˜s, a) ,(27) Reinforcement Learning Journal 2026 so: exp c(˜s) = 1 P a π(a|s) exp 1 β eA˜π(˜s, a) .(28) Substituting back into (26): ˜π′(a|˜s) = π(a|s) exp 1 β eA˜π(˜s, a) P a′ π(a′|s) exp 1 β eA˜π(˜s, a′) .(29) Now we replace the advantage eA˜πwith the Q-function eQ˜π. Since eA˜π(˜s, a) =eQ˜π(˜s, ...

2026

[2] [7]

Out-of- distribution detection for reinforcement learning agents with probabilistic dynamics models

Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann. Out-of- distribution detection for reinforcement learning agents with probabilistic dynamics models. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Sys- tems, pp. 851–859,

2023

[3] [9]

Llm post-training: A deep dive into reasoning large language models.arXiv preprint arXiv:2502.21321,

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models.arXiv preprint arXiv:2502.21321,

Pith/arXiv arXiv

[4] [12]

Scalable multi-objective reinforcement learning with fairness guarantees using lorenz dominance

Dimitris Michailidis, Willem Röpke, Diederik M Roijers, Sennay Ghebreab, and Fernando P Santos. Scalable multi-objective reinforcement learning with fairness guarantees using lorenz dominance. arXiv preprint arXiv:2411.18195,

arXiv

[5] [19]

Steering llm reasoning through bias-only adaptation

Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov, Boris Shaposhnikov, Nikita Balagansky, and Daniil Gavrilov. Steering llm reasoning through bias-only adaptation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 9213–9222,

2025

[6] [21]

Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,

Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,

Pith/arXiv arXiv

[7] [23]

A Theoretical Analysis of Inference-Time Fairness Alignment In this appendix, we provide the complete theoretical analysis of our inference-time fairness align- ment framework

Reinforcement Learning Journal 2026 Supplementary Materials The following content was not necessarily subject to peer review. A Theoretical Analysis of Inference-Time Fairness Alignment In this appendix, we provide the complete theoretical analysis of our inference-time fairness align- ment framework. For clarity and self-containment, we recall the core p...

2026

[8] [24]

The lifting gives˜π(at |˜st) =π(a t |s t), and ePmarginalises toP(s ′ |s, a)since theR-update is deterministic

QT−1 t=0 π(at |s t)P(s t+1 |s t, at). The lifting gives˜π(at |˜st) =π(a t |s t), and ePmarginalises toP(s ′ |s, a)since theR-update is deterministic. A direct and powerful consequence of Proposition A.2 is that rollouts collected by the base policy can bedirectly reusedfor offline training in fM. We simply annotate each transition with the cumula- tive re...

2026

[9] [27]

In practice,α >0and the constraint is active

deter- minesα: eitherα= 0(the unconstrained optimum already satisfiesKL≤ε) or the KL constraint binds (equality holds) andα >0. In practice,α >0and the constraint is active. Determiningβ.The parameterβ=α >0is implicitly defined by the constraint: E˜s∼d˜π KL ˜π′ β(·|˜s) ˜π(·|˜s) =ε,(31) where˜π′ β is the solution (30) parameterised byβ. The left-hand side ...

2026

[10] [28]

To ensure robust performance and a fair baseline comparison, we tuned the hyperparam- eters for these agents using the Optuna optimization framework. For the multi-objective baselines that optimize the welfare function directly during training (i.e., GGF-PPO, GGF-A2C, and GGF- DQN), we adopted the author-provided hyperparameters detailed in (Siddique et a...

2020

[11] [1998]

Do- main randomization for transferring deep neural networks from simulation to the real world

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23–30. IEEE,

2017

[12] [2002]

applied to fM, we can express the welfare gap aseJ(˜π′)− eJ(˜π) = TE ˜s∼d˜π′ h eA˜π(˜s, a) i . Sinced ˜π′ depends on the unknown policy, we replace it withd ˜πto obtain a tractable welfare surrogate that shares the same gradient at˜π ′ = ˜πand provides a local lower bound on the true improvement under a KL penalty (Schulman et al., 2015). Therefore, to ma...

2015

[13] [2004]

Aligning machiavellian agents: Behavior steering via test-time policy shaping.arXiv preprint arXiv:2511.11551,

Dena Mujtaba, Brian Hu, Anthony Hoogs, and Arslan Basharat. Aligning machiavellian agents: Behavior steering via test-time policy shaping.arXiv preprint arXiv:2511.11551,

arXiv

[14] [2010]

Decaf: Learning to be fair in multi-agent resource allocation

Ashwin Kumar and William Yeoh. Decaf: Learning to be fair in multi-agent resource allocation. arXiv preprint arXiv:2502.04281,

Pith/arXiv arXiv

[15] [2012]

Welfare and fairness in multi-objective reinforcement learning.arXiv preprint arXiv:2212.01382,

Zimeng Fan, Nianli Peng, Muhang Tian, and Brandon Fain. Welfare and fairness in multi-objective reinforcement learning.arXiv preprint arXiv:2212.01382,

Pith/arXiv arXiv

[16] [2013]

A survey on personalized align- ment—the missing piece for large language models in real-world applications

Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey on personalized align- ment—the missing piece for large language models in real-world applications. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 5313–5333,

2025

[17] [2015]

Policy shaping: Integrating human feedback with reinforcement learning.Advances in neural information processing systems, 26,

Reinforcement Learning Journal 2026 Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning.Advances in neural information processing systems, 26,

2026

[18] [2016]

Genarm: Reward guided generation with autoregressive reward model for test-time alignment.arXiv preprint arXiv:2410.08193,

Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment.arXiv preprint arXiv:2410.08193,

Pith/arXiv arXiv

[19] [2017]

URLhttp://arxiv.org/abs/ 1707.06347. Reinforcement Learning Journal 2026 Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

Pith/arXiv arXiv 2026

[20] [2018]

Inference-time pol- icy adapters (ipa): Tailoring extreme-scale lms without fine-tuning

Policy Shaping for Fair RL Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, et al. Inference-time pol- icy adapters (ipa): Tailoring extreme-scale lms without fine-tuning. InProceedings of the 2023 conference on empirical methods in natural language ...

2023

[21] [2020]

Fairness in preference-based reinforcement learning

Umer Siddique, Abhinav Sinha, and Yongcan Cao. Fairness in preference-based reinforcement learning. InICML 2023 Workshop The Many Facets of Preference-Based Learning,

2023

[22] [2021]

Latent activation editing: Inference-time refinement of learned policies for safer multirobot navigation

Satyajeet Das, Darren Chiu, Zhehui Huang, Lars Lindemann, and Gaurav S Sukhatme. Latent activation editing: Inference-time refinement of learned policies for safer multirobot navigation. arXiv preprint arXiv:2509.20623,

Pith/arXiv arXiv

[23] [2022]

Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,

Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hut- ter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,

Pith/arXiv arXiv

[24] [2023]

Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model.arXiv preprint arXiv:2505.06274,

Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, and Ying-Cong Chen. Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model.arXiv preprint arXiv:2505.06274,

Pith/arXiv arXiv

[25] [2024]

Towards fair and efficient policy learning in cooper- ative multi-agent reinforcement learning

Umer Siddique, Peilang Li, and Yongcan Cao. Towards fair and efficient policy learning in cooper- ative multi-agent reinforcement learning. InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 2744–2746, 2025a. Umer Siddique, Peilang Li, and Yongcan Cao. Learning fair pareto-optimal policies in multi- objecti...

Pith/arXiv arXiv

[26] [2025]

Optimizing generalized gini indices for fairness in rankings.arXiv preprint arXiv:2204.06521,

Virginie Do and Nicolas Usunier. Optimizing generalized gini indices for fairness in rankings.arXiv preprint arXiv:2204.06521,

Pith/arXiv arXiv

[27] [2026]

Shifting perspectives: Steer- ing vectors for robust bias mitigation in llms.arXiv preprint arXiv:2503.05371, 2025b

Zara Siddique, Irtaza Khalid, Liam D Turner, and Luis Espinosa-Anke. Shifting perspectives: Steer- ing vectors for robust bias mitigation in llms.arXiv preprint arXiv:2503.05371, 2025b. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Maste...

arXiv