HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning

Adam Sobey; Federico Cerutti; Kryspin Varys; Timothy J. Norman

REVIEW 4 major objections 4 minor 34 references

HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A scalar reputation signal, computed from each action's distance to explicit safety rules and learned social preferences, is claimed to steer reinforcement learners into value-aligned behaviour, statistically indistinguishable from human…

desk verdict A clear modular method for combining rule-based and learned norms, but the headline result rests on a circular evaluation that uses the same simulated trajectories to train the norm model and as the benchmark. read the letter →

arxiv 2505.15011 v1 pith:GB5DUKUG submitted 2025-05-21 cs.AI

Kryspin Varys , Federico Cerutti , Adam Sobey , Timothy J. Norman This is my paper

classification cs.AI

keywords ValueAlignmentRewardShapingReinforcementLearningReputationWeightingHybridNormsSocialRule-BasedTrafficJunction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes HAVA, a mechanism for making a reinforcement-learning agent follow two kinds of norms at once: hard safety and legal rules, encoded as a permitted-action set, and soft social preferences, learned from human trajectories. The agent's actions are checked against both norm sources, a scalar 'reputation' tracks how well recent behaviour complied, and every task reward is weighed by that reputation. The central claim is that this weighing shifts optimal behaviour toward value-aligned policies without hand-designing a normative reward, and that in a continuous-state traffic-junction task the HAVA policy with the slow-forgiveness setting is statistically indistinguishable from human trajectories (two-sample KS p = 0.42), while using either norm source alone fails. If this is right, it provides a way to combine verifiable rules with unverifiable learned social norms in one RL system.

What carries the argument

The mechanism does two things at once: it separates norm sources into the Alignment Value $\langle \mathit{RB}, \mathit{DD}\rangle$, where $\mathit{RB}$ is a rule-based function returning permitted actions and $\mathit{DD}$ is a learned function returning socially preferred actions, and it couples them to the MDP through a reputation state. The reputation $w_t\in[0,1]$ is the central object: it summarises the agent's normative history, falls to $\delta_t$ on the worst of the two alignment scores, and recovers slowly or quickly according to $\alpha$, which the paper interprets as a forgiveness rate. Because the reward seen by the agent is $R_{\mathrm{AV}}=w_{t+1}r_t$ for positive $r_t$ and $r_t[1+(1-w_{t+1})]$ for negative $r_t$, the same task reward function is preserved while the agent is incentivised to avoid dips in reputation. In the grid-world example this makes the norm-compliant path optimal once $\alpha$ is small enough, and in the junction task it produces a policy that respects the car-following safety dynamics and stays within the learned human speed range.

What would settle it

Run the junction experiment again with a held-out split of the human trajectories, or with real human driving logs, and recompute the two-sample KS test between HAVA's $\alpha=0.1$ policy and those held-out trajectories; if the $p$-value falls below $0.05$, the claim that HAVA 'passes for a human' is false rather than a property of the method.

Watch

Extended reading notes

Core claim

On the paper's own terms, the finding is that value alignment can be decomposed into a hard-constraint channel and a soft-preference channel, connected by a scalar reputation dynamic. Rule-based norms $\mathit{RB}:\mathcal{S}\to 2^{\mathcal{A}}$ define the permitted actions and are enforced by projecting any violating action to the nearest permitted one; data-driven norms $\mathit{DD}:\mathcal{S}\to 2^{\mathcal{A}}$, learned from human trajectories, mark the tentative social preferences. At each step the method measures the action's distance to both sets, converts each distance to an alignment score $\mathit{al}(\tau,d)=\max\{(\tau-d)/\tau,0\}$, takes the minimum as $\delta_t$, and updates the reputation $w_{t+1}=\min\{w_t+w_{\mathrm{inc}}(w_t),\delta_t\}$ with $w_{\mathrm{inc}}(w)=\alpha(e^w-1)+0.001$. The task reward is then weighted by $w_{t+1}$ (positive rewards shrunk, negative rewards magnified), and the reputation is appended to the state. The central empirical result is that, at $\alpha=0.1$, the resulting junction-crossing policy finishes the road in the human range and is statistically indistinguishable from the human trajectories ($p=0.42$), whereas a rule-based-only agent drives aggressively but legally and a data-driven-only agent violates safety norms by crossing without yielding.

Load-bearing premise

The load-bearing premise is that the simulated human trajectories used to train the learned social-norm model are a trustworthy stand-in for real human social norms and a fair benchmark for value-aligned behaviour, and the paper does not report a held-out split between those training trajectories and the ones used for evaluation.

Editorial extensions

If this is right

Value alignment can be added to an existing RL agent without redesigning the task reward: only a reputation scalar and its update rule are needed.
Rule-based norms and learned norms no longer have to share a representation; one can be a logical or rule-based system and the other a neural network.
The hyperparameter $\alpha$ gives a monotone search: once a low enough value makes the compliant policy optimal, decreasing it further does not change that optimality, so the user can stop tuning.
A purely data-driven agent cannot be relied on to respect norm severity, because without the hard $\mathit{RB}$ constraint it may trade safety violations for task reward.
A value-aligned policy can be operationally checked by asking whether its trajectories are statistically indistinguishable from a dataset of human behaviour.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves the dataset and evaluation on the same simulated human trajectories; a natural next step would be to test HAVA on a held-out split of those trajectories or on real human driving logs, to separate genuine alignment from overfitting the training data.
The reputation mechanism is representation-agnostic, so the same 'hard constraint plus learned preference plus scalar weighing' pattern could be applied in other partially normative domains, such as language agents with constitutional rules and preference data.
The exponential forgiveness schedule is a design choice; any monotone recovery function would likely give the same qualitative behaviour, which suggests automatic tuning of $\alpha$ could be done by measuring the resulting policy's distance to human data.
Because the statistical-indistinguishability result rests on a single experimental setting and hyperparameter configuration, showing the same $p>0.05$ pattern across multiple seeds and junctions would determine how general the finding is.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

HAVA is a clear, modular proposal for combining rule-based safety norms with learned social preferences through a reputation-weighted reward. The core mechanism—Eqs. 2–3, where a scalar reputation rises slowly after violations and is capped by the current alignment value—is intuitive and clean. Keeping RB and DD as separate representations with a hard safety filter on RB is a sensible design, and the ablations against RB-only and DD-only make the intended point: one is unsocial, the other can violate safety norms. The paper is also honest about its limitations, and the toy example is a transparent hand computation, even though it is not a learned result.

The soft spot is exactly where the stress-test note lands. Section 3.2.1 states that DD is trained on simulated human trajectories (dataset D). Section 3.2.4 reports a KS test comparing HAVA trajectories to the 'Human' column—which appears to be the same D. No held-out split is described. Since Eq. 4 rewards closeness to DD's predicted speed range, and DD was fit to D, the headline p=0.42 is largely a check that the agent optimizes the learned norm model, not evidence of alignment with independent social norms. This circularity is reinforced by the paper's own definition of value alignment in Section 2.3, where statistical indistinguishability from D is the goal. That is a real flaw in the empirical claim, but it is fixable. The qualitative behavior (waiting at the junction, finish times near human range) is consistent with the mechanism, so I would not dismiss the method itself.

Minor issues: no seeds or variance for the RL runs, no baseline like a direct norm penalty or constrained RL, and the code link has no commit hash. These are easy to address.

Recommendation: this deserves peer review, but with a clear demand for a held-out evaluation, variance reporting, and at least one strong baseline. The method is plausible and the paper is readable; the evaluation just needs to match the strength of the claim.

Referee Report

4 major / 4 minor

Summary. The paper proposes HAVA, a reward-shaping method for reinforcement learning that combines a rule-based norm component RB (mandatory safety/legal norms) with a data-driven component DD (tentative social norms). At each time step HAVA computes the action's distances to the permitted action sets, converts them to alignment scores using Eq. (1), updates a reputation w_t using Eqs. (2)-(3), and weights the task reward with Eq. (4). Section 2.3 defines a policy as value-aligned when its trajectories are statistically indistinguishable from the dataset D while respecting RB. The method is demonstrated on a discrete grid world and a continuous-state SUMO junction scenario, with the headline result being a 2-sample KS p-value of 0.42 between the HAVA policy (alpha=0.1) and the 'Human' trajectories, plus ablations against rule-based-only and data-driven-only variants.

Significance. If the empirical validation held up, HAVA would be a useful contribution: it offers a modular way to combine explicit and learned norm representations, it makes norm severity explicit by keeping RB mandatory while making DD violable, and the reputation mechanism is simple and interpretable. The released code and demo are additional strengths. However, the current experiments do not yet establish the paper's central claim of value-aligned behavior, because the evaluation design conflates the training signal for DD with the benchmark used to test value alignment.

major comments (4)

[3.2.1, 3.2.4, Table 2] The headline p=0.42 is a circular result. DD is trained by supervised learning on the dataset of simulated human trajectories D (Section 3.2.1), and Section 3.2.4 uses those same trajectories as the 'Human' benchmark in the KS test. Since Eq. (4) rewards the agent for staying close to DD's predicted speed range, and DD was fitted to D, HAVA is effectively optimized to reproduce the training distribution. A KS test against D is therefore a self-consistency check rather than evidence of alignment with independent human social norms, and it does not support the Section 6 claim that HAVA 'passes for a human'. The authors should evaluate on a held-out partition of D that was not used to train DD, and ideally on independently collected or independently simulated human trajectories.
[3.2.4, Table 2] No measure of variability or number of independent runs is reported. The KS comparison uses '10 policies from the last 500 training episodes' from what appears to be a single training run, and the p-values for the other comparisons are astronomically small, suggesting large effective sample sizes, but the reader cannot assess whether p=0.42 is stable across seeds or how the 10 policies were selected. The authors should report multiple seeds, confidence intervals or standard errors for the KS p-values, and the exact KS implementation, including the number of trajectories and the pooling procedure.
[2.3] The formal definition of value alignment as 'p-value > 0.05' is statistically problematic: failure to reject the null hypothesis that two distributions are equal is not evidence that they are equivalent, especially with small samples or highly variable policies. If the authors retain this definition, they should justify it and supplement it with effect sizes or equivalence tests; otherwise the phrase 'statistically indistinguishable' overstates what a KS test alone can establish.
[3.2.1, Section 6] The benchmark D is explicitly generated from the Krauss model with varied parameters, not collected from human drivers. Therefore even a non-circular KS match to D would demonstrate agreement with a synthetic driver model, not with actual human social behavior. The 'Human' column in Table 2 and the conclusion that HAVA produces policies that 'pass for a human' should be reworded, or the experiments should be supplemented with real human driving data.

minor comments (4)

[Abstract] The sentence 'We carry out a two experiments' should be corrected to 'We carry out two experiments'.
[3.1] The numerical example with alpha=10 appears to use a discount factor of gamma=0.99, but the text writes '0.995' and the expression '0.26 * 100 * 0.995 = 24.72' is arithmetically inconsistent; please correct the discount factor and the displayed arithmetic.
[5, 6, 4] There are several typos that should be fixed: 'interpratibility' in Section 5, 'unaccaptable' in Section 6, 'settting' in Section 4, and 'MDPM' in Section 2.1.
[References] Reference [27] is malformed ('Daniel Kasenberg Thomas Arnold') and should be corrected to list the authors properly.

Circularity Check

2 steps flagged · score 7.0 of 10

The headline p=0.42 is circular: dataset D is simultaneously the training set for the social-norm model DD, the formal definition of value alignment, and the 'Human' KS-test benchmark, so HAVA is rewarded to match its own training target.

self definitional [Section 2.3 (Definition 1 and value-alignment definition) combined with Section 3.2.4 (KS results)]
"its produced trajectories σ are statistically indistinguishable (p-value > 0.05) from the dataset D for any states s∈S where RB(s)∩DD(s)≠∅. ... This policy is also statistically indistinguishable from the human behaviours according to the 2-sample KS test as seen in Table 2."

The paper defines 'value aligned' as statistically indistinguishable from dataset D. It then trains DD on D (Section 3.2.1) and uses HAVA's reputation-based reward weighting to push the agent toward DD's predicted speed range. Table 2's KS test compares HAVA trajectories with the 'Human' column, which is the same simulated dataset D used for training; no held-out split or separate human dataset is described. The p=0.42 therefore checks how well the agent reproduced its own training distribution, not whether the policy matches independent human social norms.
fitted input called prediction [Section 3.2.1, Figure 5 caption; Section 3.2.4/Table 2]
"To implement the social norms captured in DD we collected a dataset of simulated human behaviours. These human behaviours follow the Krauss model at different maximal speeds and accelerations. ... We used supervised learning to train DD to solve a regression problem of predicting a minimal and maximal speed. The simulated human trajectories used for training as well as the resulting DD are visualized in Figure 5."

The only 'human' trajectories introduced in the paper are the simulated trajectories used to train DD, as the Figure 5 caption states explicitly. The KS test in Table 2 compares HAVA trajectories to a 'Human' column, but the paper never describes an independent, held-out, or real-human evaluation set. The headline claim that HAVA 'passes for a human' is therefore evaluated against the exact data that fitted DD; the outcome is a measure of how well the reward shaping imitated its own teacher, not a validated prediction about human-like value alignment.

full rationale

The paper's central empirical claim is that HAVA with alpha=0.1 produces a policy statistically indistinguishable from human trajectories (p=0.42, Table 2). The formal definition of value alignment in Section 2.3 is statistical indistinguishability from dataset D. Section 3.2.1 says DD was trained by supervised learning on 'a dataset of simulated human behaviours', and Figure 5 labels these trajectories as 'used for training'. Equations 1-4 reward the agent for staying close to DD's predicted speed range, so HAVA is optimized to reproduce D's distribution. Table 2 then performs a 2-sample KS test against a 'Human' column, with no held-out split or separately collected human dataset described anywhere in the paper. Thus the p=0.42 result confirms that the agent learned to imitate its own training distribution; it does not independently validate alignment with human social norms. The comparisons against the RB and DD ablations do illustrate the mechanism, but the headline 'pass for a human' claim is circular because the norm model's training data and the evaluation oracle are the same set. This is not primarily a self-citation concern; it is an evaluation-set identity issue: D is used as training signal, as the definition of success, and as the benchmark.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a hand-defined value-alignment criterion, a learned norm model trained on simulated data, and a reputation dynamics specified by the authors. The most consequential burden is that the same dataset D both trains DD and defines success, creating circularity. The reputation variable and the alignment definition are introduced specifically for this paper and have no external evidence.

free parameters (3)

alpha (forgiveness speed) = 0.1 for junction; searched values 10 to 1 in toy
Controls how many norm-respecting actions are needed to restore reputation; selected by scanning values until the resulting policy matched human trajectories (Sections 3.1, 3.2.4).
tau (tolerance margin) = 1 km/h
Set by appeal to Shaw et al.'s human-equivalence standard rather than derived; in continuous action space it determines when reputation drops to zero (Section 3.2.1).
DD model parameters = not specified
The data-driven norm predictor is trained on simulated human trajectories (Section 3.2.1); its accuracy is not reported against held-out data.

assumptions (5)

ad hoc to paper Value alignment is defined as statistical indistinguishability from the dataset D (Section 2.3).
This definition makes the evaluation depend on the same data that trains DD, so it is a design choice rather than an independently established criterion.
domain assumption DD trained on simulated Krauss-model driver trajectories approximates real social driving norms.
The paper uses simulated humans, not real human driving data, and never validates DD on a held-out set (Section 3.2.1).
domain assumption The Krauss car-following model plus a 50 km/h speed limit fully captures the mandatory safety and legal norms in the junction.
The claim that HAVA never violates mandatory norms rests on the simulator's model being a complete and correct representation of safety and law (Sections 3.2.1 and 3.2.5).
ad hoc to paper The reputation update equations (Eqs. 2-3) implement a sensible forgiveness dynamic.
These equations are introduced as a 'metaphor for forgiveness' and are not derived from any principle or benchmark (Section 2.2).
domain assumption The 2-sample KS test applied to the selected trajectories is an appropriate measure of policy similarity.
The test is used to claim indistinguishability, but the paper does not specify the sample sizes, the exact trajectories tested, or whether the comparison is to training or held-out data (Table 2).

invented entities (1)

reputation w_t
purpose: State variable between 0 and 1 that summarizes recent norm compliance and scales the task reward (Eqs. 2-4).
Introduced by the paper; its dynamics are fixed by the authors' equations, and there is no external measurement or falsifiable handle outside the algorithm itself.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning." pith.science (2026). https://pith.science/paper/GB5DUKUG

@misc{pith2026250515011,
  author       = {Pith},
  title        = {Pith review of: HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GB5DUKUG}},
  note         = {Machine review of arXiv:2505.15011}
}

read the original abstract

Our society is governed by a set of norms which together bring about the values we cherish such as safety, fairness or trustworthiness. The goal of value-alignment is to create agents that not only do their tasks but through their behaviours also promote these values. Many of the norms are written as laws or rules (legal / safety norms) but even more remain unwritten (social norms). Furthermore, the techniques used to represent these norms also differ. Safety / legal norms are often represented explicitly, for example, in some logical language while social norms are typically learned and remain hidden in the parameter space of a neural network. There is a lack of approaches in the literature that could combine these various norm representations into a single algorithm. We propose a novel method that integrates these norms into the reinforcement learning process. Our method monitors the agent's compliance with the given norms and summarizes it in a quantity we call the agent's reputation. This quantity is used to weigh the received rewards to motivate the agent to become value-aligned. We carry out a series of experiments including a continuous state space traffic problem to demonstrate the importance of the written and unwritten norms and show how our method can find the value-aligned policies. Furthermore, we carry out ablations to demonstrate why it is better to combine these two groups of norms rather than using either separately.

Figures

Figures reproduced from arXiv: 2505.15011 by the authors.

**Figure 1.** At each time-step Alignment Value AV receives 𝑠𝑡 , 𝑎𝑡 from the agent and calculates the new agent’s reputation 𝑤𝑡+1 which then becomes part of the state 𝑠𝑡+1. An agent’s action is sent to the environment only if it is within the permitted rule-based actions. If not, another (closest) rulebased action is executed instead. 2 HYBRID APPROACH TO VALUE ALIGNMENT 2.1 Markov Decision Processes and Reinforcement Learning M… view at source ↗

**Figure 2.** Development of two HAVA agents’ reputation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Toy Example: Grid world with three policies [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Simulated human trajectories used to train [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 4.** Figure 4: The agent is spawned and must learn how to cross [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 6.** Figure 6: We see the three strategies that the pure data [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: HAVA produces two clusters of policies. While 𝛼 of 10 (4 steps) and 0.5 (15 steps) does not seem to be able to converge to a human-like behaviour, setting 𝛼 = 0.1 (45 steps) produces a value aligned policy [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 22 canonical work pages

[1]

Shani Alkoby, Avilash Rath, and Peter Stone. 2019. Teaching Social Behavior through Human Reinforcement for Ad hoc Teamwork - The STAR Framework: Extended Abstract. InProceedings of the 18th International Conf. on Autonomous Agents and MultiAgent Systems(Montreal QC, Canada)(AAMAS ’19). Interna- tional Foundation for Autonomous Agents and Multiagent Syste...

work page 2019
[2]

Colin Allen, Iva Smit, and Wendell Wallach. 2005. Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches.Ethics and Information Technology7, 3 (01 Sep 2005), 149–155. https://doi.org/10.1007/s10676-006-0004-4

work page doi:10.1007/s10676-006-0004-4 2005
[3]

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018. Safe Reinforcement Learning via Shielding. Proceedings of the AAAI Conf. on Artificial Intelligence32, 1 (Apr. 2018). https: //ojs.aaai.org/index.php/AAAI/article/view/11797

work page 2018
[4]

2011.Machine Ethics

Michael Anderson and Susan Leigh Anderson (Eds.). 2011.Machine Ethics. Cam- bridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511978036

doi:10.1017/cbo9780511978036 2011
[5]

Mattijs Baert, Sam Leroux, and Pieter Simoens. 2023. Inverse reinforcement learning through logic constraint inference.Machine Learning(14 Mar 2023). https://doi.org/10.1007/s10994-023-06311-2

work page doi:10.1007/s10994-023-06311-2 2023
[6]

Avinash Balakrishnan, Djallel Bouneffouf, Nicholas Mattei, and Francesca Rossi

work page
[7]

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pe- dro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Us- man Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michau...

work page 2023
[8]

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. 2019. Noisy Networks for Exploration. arXiv:1706.10295 [cs.LG] https://arxiv.org/abs/1706.10295

arXiv 2019

Show all 34 references

[9]

Yue Guo, Boshi Wang, Dana Hughes, Michael Lewis, and Katia Sycara. 2020. Designing Context-Sensitive Norm Inverse Reinforcement Learning Framework for Norm-Compliant Autonomous Agents. In2020 29th IEEE International Conf. on Robot and Human Interactive Communication (RO-MAN). ...

2020
[11]

Daniel Kasenberg and Matthias Scheutz. 2018. Norm Conflict Resolution in Stochastic Domains.Proceedings of the AAAI Conf. on Artificial Intelligence32, 1 (Apr. 2018). https://doi.org/10.1609/aaai.v32i1.11295

2018 doi
[12]

Bettina Könighofer, Julian Rudolf, Alexander Palmisano, Martin Tappler, and Roderick Bloem. 2021. Online Shielding for Stochastic Systems. InNASA Formal Methods: 13th International Symposium, NFM 2021, Virtual Event, May 24–28, 2021, Proceedings. Springer-Verlag, Berlin, Heide...

2021
[13]

Hauptabt

Koeln (Germany). Hauptabt. Mobilitaet und Systemtechnik]" "Krauss, S. [DLR Forschungszentrum fuer Luft-und Raumfahrt e.V. 1998.Microscopic modeling of traffic flow: investigation of collision free vehicle dynamics. Technical Report. Germany. DLR Deutsches Zentrum fuer Luft- un...

1998
[14]

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. InForty-first ...

2024
[15]

Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun- Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wag- ner, and Evamarie Wießner. 2018. Microscopic Traffic Simulation using SUMO, In The 21st IEEE International Conf. on Intel...

2018
[16]

Rusu, Joel Veness, Marc G

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

2015 doi
[17]

Md Sultan Al Nahian, Spencer Frazier, Brent Harrison, and Mark O. Riedl. 2021. Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior. CoRRabs/2104.09469 (2021). arXiv:2104.09469 https://arxiv.org/abs/2104.09469

2021 arXiv
[18]

Emery A. Neufeld. 2022. Reinforcement Learning Guided by Provable Normative Compliance. InProceedings of the 14th International Conf. on Agents and Artificial Intelligence - Volume 3: ICAART,. INSTICC, SciTePress, 444–453. https://doi.org/ 10.5220/0010835600003116

2022 doi
[19]

Neufeld, Ezio Bartocci, and Agata Ciabattoni

Emery A. Neufeld, Ezio Bartocci, and Agata Ciabattoni. 2022. On Normative Re- inforcement Learning via Safe Reinforcement Learning. InPRIMA 2022: Principles and Practice of Multi-Agent Systems, Reyhan Aydoğan, Natalia Criado, Jérôme Lang, Victor Sanchez-Anguix, and Marc Serram...

2022
[20]

Neufeld, Ezio Bartocci, Agata Ciabattoni, and Guido Governatori

Emery A. Neufeld, Ezio Bartocci, Agata Ciabattoni, and Guido Governatori
[21]

Varshney, Murray Campbell, Moninder Singh, and Francesca Rossi

Ritesh Noothigattu, Djallel Bouneffouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, Kush R. Varshney, Murray Campbell, Moninder Singh, and Francesca Rossi. 2019. Teaching AI Agents Ethical Values Using Reinforcement Learning and Policy Orchestration. InProceedings of the T...

2019 doi
[22]

Osoba, Benjamin Boudreaux, and Douglas Yeung

Osonde A. Osoba, Benjamin Boudreaux, and Douglas Yeung. 2020. Steps Towards Value-Aligned Systems. InProceedings of the AAAI/ACM Conf. on AI, Ethics, and Society(New York, NY, USA)(AIES ’20). Association for Computing Machinery, New York, NY, USA, 332–336. https://doi.org/10.1...

2020
[23]

Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. 2022. ASE: large-scale reusable adversarial skill embeddings for physically simulated characters.ACM Trans. Graph.41, 4, Article 94 (July 2022), 17 pages. https: //doi.org/10.1145/3528223.3530110

2022
[24]

Oliehoek, and Luciano C

Markus Peschl, Arkady Zgonnikov, Frans A. Oliehoek, and Luciano C. Siebert
[25]

Francesca Rossi and Nicholas Mattei. 2019. Building Ethically Bounded AI. Proceedings of the AAAI Conf. on Artificial Intelligence33, 01 (Jul. 2019), 9785–

2019
[26]

Shaw, Andreas Stöckel, Ryan W

Nolan P. Shaw, Andreas Stöckel, Ryan W. Orr, Thomas F. Lidbetter, and Robin Cohen. 2018. Towards Provably Moral AI Agents in Bottom-up Learning Frame- works. InProceedings of the 2018 AAAI/ACM Conf. on AI, Ethics, and Society(New Orleans, LA, USA)(AIES ’18). Association for Co...

2018
[27]

InProceedings of the 21st International Conf

MORAL: Aligning AI with Human Norms through Multi-Objective Re- inforced Active Learning. InProceedings of the 21st International Conf. on Au- tonomous Agents and Multiagent Systems(Virtual Event, New Zealand)(AAMAS ’22). International Foundation for Autonomous Agents and Mult...
[28]

Hado van Hasselt, Arthur Guez, and David Silver. 2015. Deep Reinforcement Learning with Double Q-learning. arXiv:1509.06461 [cs.LG] https://arxiv.org/ abs/1509.06461

2015 arXiv
[29]

Czarnecki, Michaël Mathieu, An- drew Dudzik, Junyoung Chung, David H

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, An- drew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max...

2019
[30]

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. 2016. Dueling network architectures for deep reinforcement learning. InProceedings of the 33rd International Conf. on International Conf. on Machine Learning - Volume 48(New York, NY, U...

2016
[31]

Daniel Kasenberg Thomas Arnold. 2017. Value Alignment or Misalignment – What Will Keep Systems Accountable?AAAI Workshop on AI, Ethics, and Society (2017). https://par.nsf.gov/biblio/10301363

2017
[35]

Yueh-Hua Wu and Shou-De Lin. 2018. A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents. InProceedings of the Thirty-Second AAAI Conf. on AI (AAAI’18/IAAI’18/EAAI’18). AAAI Press, Article 206, 8 pages

2018
[2019]

on Artificial Intelligence33, 01 (Jul

Incorporating Behavioral Constraints in Online AI Systems.Proceedings of the AAAI Conf. on Artificial Intelligence33, 01 (Jul. 2019), 3–11. https://doi.org/ 10.1609/aaai.v33i01.33013

2019 doi
[2022]

https://doi.org/10.1007/s10676- 022-09665-8

Enforcing ethical goals over reinforcement-learning policies.Ethics and Information Technology24, 4 (29 Sep 2022), 43. https://doi.org/10.1007/s10676- 022-09665-8

2022 doi
[9789]

https://doi.org/10.1609/aaai.v33i01.33019785

doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Shani Alkoby, Avilash Rath, and Peter Stone. 2019. Teaching Social Behavior through Human Reinforcement for Ad hoc Teamwork - The STAR Framework: Extended Abstract. InProceedings of the 18th International Conf. on Autonomous Agents and MultiAgent Systems(Montreal QC, Canada)(AAMAS ’19). Interna- tional Foundation for Autonomous Agents and Multiagent Syste...

work page 2019

[2] [2]

Colin Allen, Iva Smit, and Wendell Wallach. 2005. Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches.Ethics and Information Technology7, 3 (01 Sep 2005), 149–155. https://doi.org/10.1007/s10676-006-0004-4

work page doi:10.1007/s10676-006-0004-4 2005

[3] [3]

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018. Safe Reinforcement Learning via Shielding. Proceedings of the AAAI Conf. on Artificial Intelligence32, 1 (Apr. 2018). https: //ojs.aaai.org/index.php/AAAI/article/view/11797

work page 2018

[4] [4]

2011.Machine Ethics

Michael Anderson and Susan Leigh Anderson (Eds.). 2011.Machine Ethics. Cam- bridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511978036

doi:10.1017/cbo9780511978036 2011

[5] [5]

Mattijs Baert, Sam Leroux, and Pieter Simoens. 2023. Inverse reinforcement learning through logic constraint inference.Machine Learning(14 Mar 2023). https://doi.org/10.1007/s10994-023-06311-2

work page doi:10.1007/s10994-023-06311-2 2023

[6] [6]

Avinash Balakrishnan, Djallel Bouneffouf, Nicholas Mattei, and Francesca Rossi

work page

[7] [7]

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pe- dro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Us- man Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michau...

work page 2023

[8] [8]

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. 2019. Noisy Networks for Exploration. arXiv:1706.10295 [cs.LG] https://arxiv.org/abs/1706.10295

arXiv 2019