{"id":"f669e1e8-4772-47b3-8a5a-4d9bfa75cf20","arxiv_id":"2412.15163","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"RAWL-E adds a maximin-based reward to norm-learning agents and reports fairer, more robust simulated harvesting societies, though several of the gains are statistically weak or built into the reward itself.","lead":"This paper introduces RAWL-E, an agent design that adds a Rawlsian fairness reward to reinforcement learning so that simulated agents consider the least well-off agent when deciding how to act. The method is tested in virtual berry-harvesting worlds and is meant to show how ethical principles can be embedded into AI decision-making.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Norms module is causally inert: it only records learned behaviours and is never consulted during action selection, so the claimed mechanism that emerged norms enhance welfare and fairness is unsupported; observed gains can be fully explained by reward shaping.","rationale":"Read in good faith, RAWL·E is a transparent way to add maximin reward shaping to DQN; if the aim were simply to show that such shaping improves the minimum well-being, the experiments are suggestive. The paper, however, makes a stronger, mechanistic claim about norms. That claim is the load-bearing element because it is the stated novelty over prior Rawlsian RL work (Zimmer et al., Balakrishnan et al.) and the reason the title emphasizes norm-learning agents. The architecture makes the norm module causally irrelevant to behaviour, so the experiments cannot support the claim that norms enhance outcomes. This is not a consensus disagreement; it is internal to the algorithm description. The reader's weakest assumption (credit assignment of min-well-being changes in asynchronous settings) is real, but it attacks the quality of the ethical reward signal, not the claimed norm mechanism. The norm-module issue is more decisive: even a perfectly attributed maximin reward would not make norms causally effective, because norms are never consulted. The well-being formula inconsistency (Eq. 4 divides by negative hdecay) is a separate soundness issue but not needed for this objection. I recommend CONDITIONAL rather than REJECT because the empirical observations about reward shaping could be salvaged by honest rewording and an ablation; however, the conditions must include removing the causal norm claims and adding the ablation.","tokens_in":15617,"tokens_out":6055,"duration_ms":58623,"concrete_test":"Run the RAWL·E society with NormsModule disabled (remove Algorithm 2 calls, keep the EM reward shaping and all other settings identical). If metrics M1–M4 are statistically indistinguishable from full RAWL·E, the norms module is causally inert and the central mechanism claim fails. A complementary check: apply the same Ft+1 shaping to the baseline DQN agents; if baseline with shaping matches RAWL·E, the entire behavioural difference is reward shaping, not norm learning.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 1) is that RAWL·E agents 'learn ethical norms that promote the well-being of the least advantaged' and that 'norms emerging in RAWL·E agent societies enhance social welfare, fairness, and robustness.' The architecture does not support this causal story. In Algorithm 3, the policy π selects at from st (line 1), the shaped reward r′t+1 = rt+1 + Ft+1 is used to update Π(θ,a) (lines 5–7), and only afterwards are νt, at, r′t+1 passed to NormsModule (lines 8–9). Nothing from NormsModule or the norm base is ever fed back into action selection; the norm base is written but never read by the decision process. Norms are therefore emergent statistics of behaviour, not causes of it. The experimental comparison (Section 5) contrasts full RAWL·E (ethics reward shaping plus norms recording) against baseline DQN; since the norms module cannot influence behaviour, any differences in M1–M4 are attributable to the shaped reward Ft+1, not to norms. The hypotheses H1–H4 are phrased as 'norms emerging ... lead to' outcomes, which is unsupported. If the claim is weakened to 'recorded norms correlate with outcomes,' that is not the stated contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAWL·E, a method that augments DQN agents in a multi-agent harvesting domain with a Rawlsian maximin-based ethics module. The ethics module computes a self-directed sanction from the change in minimum well-being between time steps and adds it to the environmental reward. A separate norms module records behaviors and detects when 90% of agents share a behavior, treating that as an emerged norm. The authors evaluate RAWL·E against a baseline DQN society in two harvesting scenarios (capabilities and allotment), reporting that RAWL·E societies have lower inequality, higher minimum experience, higher social welfare, and higher robustness. They conclude that norms emerging in RAWL·E societies lead to fairer and more sustainable collective behavior.","tokens_in":15850,"tokens_out":4587,"duration_ms":34655,"significance":"If the causal claims about norm emergence were empirically supported, this paper would make a useful contribution by connecting normative ethics (Rawlsian maximin) to bottom-up norm emergence in multi-agent reinforcement learning. Strengths include a publicly released codebase, detailed parameter tables in appendices, and a clearly described modular architecture. However, the significance is substantially weakened by three issues: (1) a sign error in the well-being formula that likely corrupts the ethics signal and the main fairness metric; (2) a circularity between the reward function and the primary evaluation metric; and (3) an architecture in which the norms module is never consulted during action selection, so the observed differences are attributable to reward shaping rather than to norms. The statistical claims also overreach the reported data.","major_comments":[{"comment":"The well-being formula is agwell-being = aghealth + (agberries × hgain) / hdecay. Since hgain = 0.1 and hdecay = −0.01, the second term equals −10 × agberries, so well-being decreases with berry count. This sign error directly affects the ethics module's input (the minimum well-being in Eq. (1)) and the M2 metric (minimum experience) that is used to support H2. The formula should be corrected (e.g., hgain × agberries / |hdecay|, or hgain × agberries plus health) and the experiments re-run, as the reported numeric values and effect sizes may change substantially.","section":"Eq. (4), Section 4.1"},{"comment":"The evaluation metric M2 (minimum experience) is the same quantity that the ethics module rewards: Eq. (2) gives a positive sanction when the minimum well-being increases, and M2 measures the minimum well-being across agents. Therefore the finding that RAWL·E societies have higher minimum experience is not an emergent property of norms; it is a direct consequence of the reward shaping. The claim in H2 and in the abstract that norms 'lead to' higher minimum experience is thus circular. The paper needs a metric that is not the optimization target of the sanction, or an explicit argument that the improvement exceeds what reward shaping alone would produce.","section":"Sections 3.2 and 4.4"},{"comment":"The norms module is causally inert with respect to behavior. In Algorithm 3, the policy is updated using the shaped reward (lines 5–7), and only afterward are νt, at, and r′t+1 passed to the norms module (lines 8–9). Nothing from the norms module or the norm base is fed back into action selection. Thus the norms module records behavior but never influences it. Consequently, any differences in M1–M4 between RAWL·E and baseline agents are attributable to the ethics-module reward Ft+1, not to emerged norms. The hypotheses H1–H4 are phrased as 'norms emerging ... lead to' outcomes, which is not supported by this architecture. An ablation separating reward shaping from norm recording, or a mechanism by which norms affect decisions, is needed to support the stated contribution.","section":"Algorithm 3, Section 3.2"},{"comment":"In the asynchronous setting, agents act in random order within each step. The ethics module compares Ut and Ut+1, where the minimum well-being can change because of another agent's action in the same step. The sanction Ft+1 is then assigned to the acting agent, even if the change in the minimum was caused by a different agent. This attribution problem means the learning signal may reward or punish an agent for effects it did not cause, undermining the interpretation that RAWL·E agents 'learn ethical behavior' rather than merely responding to a noisy proxy. The paper should either restrict the comparison to changes caused by the acting agent or analyze the extent of misattribution.","section":"Section 4.1, Eq. (2)"},{"comment":"The headline claims overreach the reported statistics. For the M2 (minimum experience) metric on agresource, the differences are not significant in either scenario (d = 0.15 in capabilities, d = 0.27 in allotment; the text says p > 0.01). For M3 (social welfare) on agresource, differences are not significant (d = 0.04 and 0.14). For M4 (robustness), although p < 0.01, the effect sizes are negligible (d = 0.18 and 0.11). Yet the Summary of Findings states 'Our results support our hypotheses' and the abstract claims 'higher social welfare, fairness, and robustness' without these qualifications. The conclusions should be scaled back to the metrics that show significant and at least small-to-medium effects, or the hypotheses should be revised to match the evidence.","section":"Section 5.2, Table 8"}],"minor_comments":[{"comment":"The hypothesis labels are inconsistent: Section 4.4 defines H1 as minimum experience and H2 as inequality, but Section 5.2 presents 'H1 (inequality)' and 'H2 (minimum experience)'. The numbering should be corrected throughout.","section":"Section 4.4"},{"comment":"Algorithm 2 uses 'clipNorm' (line 8) and updates emerged norms (line 10), but Table 1 lists 'tclip behaviours' and 'tclip norms' without explaining how these map to clipNorm and the clipping interval. The relation between the parameters and the algorithm should be made explicit.","section":"Algorithm 2 and Table 1"},{"comment":"Tables 3 and 8 contain overlapping data for the allotment scenario; Table 8 in the appendix duplicates most of Table 3. Either merge them or clearly separate the summary table from the full appendix table to avoid confusion.","section":"Tables 3 and 8"},{"comment":"The captions for Figures 3 and 4 say the quantities are 'summed for e, normalised by step frequency', but the y-axis labels are just 'Minimum agwell-being' and 'Cumulative agwell-being'. Please define the normalization and the aggregation in the caption or in the text so the reader can interpret the plots accurately.","section":"Figures 3 and 4"},{"comment":"The paper uses 'RAWL·E' with a middle dot in most of the text, but the abstract, headings, and some equations sometimes use 'RAWL-E' or 'RAWL·E' inconsistently. Please standardize the notation.","section":"Notation"},{"comment":"The reward normalization is described as giving RAWL·E agents 'lower raw rewards', but Table 6 shows that for 'Try to eat without berries' and 'Try to throw without berries' the RAWL·E penalty is −0.10 while baseline is −0.20. Please clarify the normalization procedure, since the stated rationale does not match all entries.","section":"Section 4.3, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a publicly available codebase and a clear presentation of the architecture, but the technical issues are substantial. The sign error in Eq. (4) suggests the reported numbers may not be reliable; the circularity of M2 with the reward function means the fairness claim is largely by construction; and the norms module is not causally connected to behavior, making the central 'norms lead to' claim unsupported. These are fixable in principle—by correcting the formula, adding an ablation without the norms module, and reframing the claims to be about the ethics-module reward shaping rather than norm emergence—but they require additional experiments and careful rewriting. I would support a major-revision decision rather than rejection, because the core idea is potentially valuable and the authors have provided a solid experimental infrastructure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the norms module does not feed back into action selection. Algorithm 3 writes to the norm base but nothing ever reads from it into the policy. So the differences between RAWL·E and baseline—where they exist—are wholly attributable to the shaped reward, not to 'norms emerging.' The hypotheses H1–H4 are phrased as norms causing outcomes, but the architecture cannot support that.\n\nWhat's actually new and useful: applying maximin reward shaping to DQN agents in a harvesting game, with a transparent implementation and public code. That is a legitimate small contribution. The reward normalization between baseline and RAWL·E is a fair attempt to isolate the ethics signal, and the appendices are unusually thorough.\n\nThe soft spots are substantial. First, the metric M2 (minimum experience) is exactly the quantity the ethics module rewards. A positive sanction is given when the minimum increases, so RAWL·E societies having higher M2 is by construction, not a discovery. Second, Eq. 4 defines well-being as health + (berries × hgain)/hdecay; since hdecay is negative, more berries make well-being lower. That sign error is hard to reconcile with the text. Third, Section 4.3 describes a sanction for 'could have improved but did not,' but Algorithm 1 only checks whether the actual minimum changed; the implemented sanction is simpler than described. Fourth, for agresource, the minimum-experience and social-welfare differences are not significant, and robustness effects are negligible (d = 0.11–0.18), so the headline overreaches even on its own terms.\n\nThe paper is a decent engineering write-up of a reward-shaping baseline, with good reproducibility. It is not a demonstration of norm emergence influencing behavior; the norm module is an inert record-keeper. For a reader interested in fairness rewards in MARL, it's worth a skim. For a reader hoping to see norms do causal work, it will disappoint.\n\nI would send it to peer review only with the expectation of major revision: rephrase the claims to say the shaped reward improves fairness, remove the norm-emergence causation language, fix Eq. 4, and reconcile the sanction description. With those changes it could be a solid workshop-level paper; as is, the central claim is unsupported.","headline":"Maximin reward shaping works in a simple grid world, but the norm module is decorative and the paper's causal claims about norms are not supported by its own architecture.","tokens_in":16421,"tokens_out":3444,"would_cite":false,"duration_ms":29524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAWL·E embeds Rawlsian maximin into each agent's reward, producing fairer norms in simulated societies.","keywords":["norm emergence","multi-agent reinforcement learning","Rawlsian ethics","maximin","fairness","reward shaping","social welfare","DQN"],"falsifier":"Run the same harvest scenarios with an oracle that attributes the minimum-well-being change to its true cause through counterfactual rewards computed by replaying the step without the agent's action, and compare to RAWL·E; if the fair-metrics advantage disappears or reverses, the observed fairness improvement is an artefact of misattributed sanction rather than learned ethics. A cheaper check is to log per-step cases where the acting agent's action could not have changed the minimum yet the sanction was nonzero, and test whether those cases alone explain the inequality reduction.","tokens_in":1740,"feed_emoji":"🤝","tokens_out":3799,"duration_ms":64950,"temperature":0.7,"pith_summary":"This paper sets out to show that a multi-agent reinforcement learning system can be made to learn ethical social norms by building a Rawlsian maximin principle directly into each agent's reward. The authors propose RAWL·E, in which an ethics module compares the lowest well-being in the society before and after an agent acts, and rewards or penalises the agent according to whether that minimum rose, stayed flat, or fell. They test the method in two simulated berry-harvesting worlds, comparing societies of RAWL·E agents to baseline deep-Q agents whose rewards are normalised to be lower. They report that RAWL·E societies produce more cooperative and more generalised norms, lower Gini inequality, higher minimum well-being, and longer survival. The significance is that it offers a way to derive 'ought' from a normative principle rather than from observed behaviour, which is what they argue lets it avoid perpetuating bad existing norms.","feed_headline":"Rawlsian reward shaping yields fairer norm-learning agents","feed_subtitle":"In simulated harvests, agents that maximize the minimum well-being learn more cooperative norms and survive longer.","key_machinery":"The carrying mechanism is the ethics module that implements the maximin function $M_A(d) = \\min_w u(d, \\upsilon_i)$ adapted from Leben, comparing the minimum well-being before and after an action and generating a self-directed sanction $F_{t+1}$ via Equation (2). This sanction is combined with the environmental reward through reward shaping, $r'_{t+1} = r_{t+1} + F_{t+1}$, so every agent's DQN policy is trained on a signal that includes whether the least advantaged agent was made better off. The norms module stores behaviours as if-then rules over preconditions and actions, counts their usage, and decays their fitness, so that the 90%-converged norms that emerge are those the society actually uses under the shaped reward.","core_discovery":"The central claim is that operationalising Rawlsian maximin in individual decision-making changes which norms emerge in a multi-agent society, and that the emerged norms are fairer and more robust. In the paper's own terms, a RAWL·E agent takes the vector of all agents' well-being $U_t$ and $U_{t+1}$, finds $\\upsilon_{\\min}$ at each step, and produces a self-directed sanction $\\xi = 0.4$ if the minimum improved, $-\\xi$ if it worsened, and 0 otherwise; this sanction is added to the environmental reward to form the learning signal. Across 2000 episodes in both harvesting scenarios, societies of RAWL·E agents showed lower inequality (Gini), higher minimum experience, higher social welfare, and longer episodes than baseline DQN societies, and the emerged cooperative norms were more generalised (for example, IF <high health> THEN <throw>). The paper states this as evidence that normative ethics can be operationalised to promote ethical norm emergence without relying on descriptive accounts of existing behaviour.","pith_inferences":["If the attribution assumption is the real driver, a counterfactual-credit variant (rewarding only when the acting agent's own action raised the minimum) would tell whether the improved metrics come from learning to help or from a noisy but benevolent signal; this is a testable extension the paper does not run.","The same reward-shaping template could be applied to other normative principles, such as egalitarian or prioritarian functions, and compared directly; the paper leaves this comparison to future work.","In environments with simultaneous moves or confounded effects, the per-step min-difference sanction may be too blunt, suggesting the method may need temporal credit assignment to scale beyond the asynchronous one-agent-at-a-time setting used here.","The generality of the emerged cooperative norms (IF high health THEN throw) suggests the shaped reward biases the society toward a division of labour, but the paper does not test whether this specialisation is robust to changes in agent capabilities or resource distribution; a follow-up with perturbed environments would clarify."],"forward_implications":["RAWL·E societies learn cooperative norms that are more generalised and used more often: for example, IF <high health> THEN <throw> emerges in the RAWL·E society while baseline cooperative norms remain more specialised.","Inequality falls: the Gini index for well-being drops from 0.20 to 0.10 in the allotment scenario, with large effect sizes (d = 1.58).","Minimum individual well-being is higher in RAWL·E societies (10.82 vs 7.18 in allotment well-being, d = 3.09), meaning the least advantaged agent is better off.","Social welfare and robustness improve: cumulative well-being is higher and episodes last longer, though the robustness effect is negligible (d = 0.11).","Because the ethics module is decoupled from the environment, the method is compatible with other RL algorithms and scenarios, so the same reward-shaping pattern can be applied elsewhere."],"supporting_citations":[{"why":"Supplies the maximin principle—maximise the minimum experience—that the ethics module operationalises.","marker":"(Rawls 1958)"},{"why":"Provides the ethical utility function and the maximin expression adapted as Equation (1).","marker":"(Leben 2020)"},{"why":"Defines the 90% convergence threshold used to decide when a behaviour counts as an emerged norm.","marker":"(Kittock 1995)"},{"why":"Gives a method for explicit norm emergence that optimises summed payoffs, which RAWL·E contrasts with by prioritising the minimum.","marker":"(Agrawal et al. 2022)"},{"why":"Provides the DQN reinforcement learning framework that the interaction module builds on.","marker":"(Sutton and Barto 2018)"},{"why":"Supplies the non-parametric significance test used to evaluate all four hypotheses.","marker":"(Mann and Whitney 1947)"},{"why":"Supplies the effect size measure d used to interpret the magnitude of differences between societies.","marker":"(Cohen 1988)"}],"fun_headline_variants":["Maximin reward shaping yields fairer norm-learning agents","Rawlsian agents learn more robust and fair norms in harvests","Sanctioning drops in minimum well-being improves agent fairness","Ethical norm emergence with Rawlsian maximin in multi-agent systems","Fairness via self-imposed sanctions in norm-learning agents"],"cache_read_input_tokens":18560,"weakest_assumption_plain":"The learning signal assumes the change in the minimum well-being from one step to the next was caused by the acting agent's own action, so sanctioning that change teaches correct ethical behaviour; in the asynchronous setting other agents' actions also move the minimum, which can reward or punish an agent for effects it did not cause.","fun_headline_variants_meta":{"raw":{"variants":["Maximin reward shaping yields fairer norm-learning agents","Rawlsian agents learn more robust and fair norms in harvests","Sanctioning drops in minimum well-being improves agent fairness","Ethical norm emergence with Rawlsian maximin in multi-agent systems","Fairness via self-imposed sanctions in norm-learning agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1352,"prompt_tokens":872,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":488,"tokens_out":480,"duration_ms":4153,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:33:51.953943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same harvest scenarios with an oracle that attributes the minimum-well-being change to its true cause through counterfactual rewards computed by replaying the step without the agent's action, and compare to RAWL·E; if the fair-metrics advantage disappears or reverses, the observed fairness improvement is an artefact of misattributed sanction rather than learned ethics. A cheaper check is to log per-step cases where the acting agent's action could not have changed the minimum yet the sanction was nonzero, and test whether those cases alone explain the inequality reduction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the maximin principle—maximise the minimum experience—that the ethics module operationalises."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ethical utility function and the maximin expression adapted as Equation (1)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 90% convergence threshold used to decide when a behaviour counts as an emerged norm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives a method for explicit norm emergence that optimises summed payoffs, which RAWL·E contrasts with by prioritising the minimum."},{"cited_title":"S.; and Barto, A","cited_arxiv_id":null,"evidence_quote":"Provides the DQN reinforcement learning framework that the interaction module builds on."},{"cited_title":"B.; and Whitney, D","cited_arxiv_id":null,"evidence_quote":"Supplies the non-parametric significance test used to evaluate all four hypotheses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the effect size measure d used to interpret the magnitude of differences between societies."}],"review_version":1}