{"id":"8ce2f2ea-40df-4ce2-a595-6d048e4fd859","arxiv_id":"2502.03953","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding retrospective and prospective reward-disparity penalties to PPO lowers demographic parity and conditional statistical parity disparities in two multi-agent simulations, at a measurable efficiency cost.","lead":"Fair-PPO adds a fairness penalty to the PPO reinforcement-learning objective, pushing agents to balance rewards between groups with and without a sensitive attribute using both past rewards and value estimates. The authors test it in a berry-collection game and a new hospital simulation, reporting lower measured disparity than plain PPO in the simpler game, with smaller and less consistent gains in the hospital.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6)'s retrospective α-term is a constant w.r.t. the policy parameters, so as written it cannot produce any policy gradient; the claimed two-component Fair-PPO mechanism is formally inert and requires an undocumented reward-shaping/λ coupling to explain the reported α-sweep results.","rationale":"The reader's conditional verdict highlights statistical robustness and overclaims, and those are legitimate concerns. But the more fundamental issue is internal to the paper's formalism: the retrospective term in Eq. (6) has zero gradient with respect to the policy parameters, because empirical episode returns are constants in the loss. This is not a question of external consensus or a missing error bar; it directly affects whether the proposed algorithm works as described. The paper's reported variation with α, including β=0 runs, is therefore not explained by the equations as written. I credit the prospective value-based penalty as a genuine design idea, and the AH experiments are suggestive; the availability of code means this concern is concretely checkable. If the code reveals reward shaping or another differentiable path for the α-term, the paper needs a corrected objective. If it does not, the α-sweep results and the two-component narrative are unsupported. Thus I recommend keeping the verdict conditional, but the condition should be a mechanism check rather than only additional error bars.","tokens_in":20584,"tokens_out":7652,"duration_ms":84122,"concrete_test":"Inspect the released repository or instrument a minimal reproduction: fix the random seed, train PPO (α=β=0) and Fair-PPO with β=0, α>0, with dynamic λ disabled; also compute the gradient ∂Lfair-DP/∂θ from a logged episode and verify whether the α-term contributes any nonzero gradient. If the gradient is zero and the β=0 run still differs from PPO, the α effect is an artifact or an undocumented mechanism; if the gradient is nonzero, identify the exact code path (e.g., reward shaping) and require it to replace or clarify Eq. (6) in the manuscript.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Eq. (6) is presented as a penalty added to the PPO loss: LFair-PPO = LCLIP - c1 LVF + c2 H - λ Lfair, with Lfair-DP_t = α|Ḡ_N1(τ) - Ḡ_N0(τ)| + β|V̄_N1(s_t) - V̄_N0(s_t)|. In any standard autodiff implementation of PPO, Ḡ_N1(τ) and Ḡ_N0(τ) are scalars computed from the rewards collected in the episode; they are not differentiable functions of the policy parameters θ. Therefore ∂/∂θ of the α-term is identically zero, and the retrospective component cannot by itself change the policy. Only the β-term, through the value-network outputs V_θ, can provide a gradient. The paper reports that varying α changes behavior, including β=0 configurations in Table 13 (e.g., α=0.25, β=0 gives disparity 6.52 vs. PPO 6.83), which cannot happen under Eq. (6) unless the implementation (a) folds the penalty into rewards or advantages before computing returns, (b) lets a 'dynamic λ' couple α to the β-term, or (c) is subject to seed noise. None of (a)-(c) is specified. This undermines the central claim that the retrospective and prospective components jointly produce the fairness spectrum, and it leaves the reported α-sweep unexplained by the paper's own equations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Fair-PPO, a modification of PPO that adds a fairness penalty, composed of a retrospective term (comparing last-episode total rewards between sensitive and non-sensitive groups) and a prospective term (comparing value-function estimates), to the PPO loss. The penalty is derived from demographic parity, with analogous penalties for counterfactual fairness and conditional statistical parity in the supplement. The method is evaluated in the Allelopathic Harvest (AH) multi-agent game and in a newly introduced HospitalSim (HS) simulation, and is compared against PPO, FEN, and SOTO. The authors report that Fair-PPO produces fairer policies than PPO, spans a spectrum of fairness-efficiency trade-offs, and that fairness comes at an efficiency cost without worsening the Gini index.","tokens_in":20818,"tokens_out":6862,"duration_ms":67414,"significance":"If the proposed mechanism is correct, Fair-PPO would be a practical, penalty-based approach for injecting group-fairness constraints into PPO for multi-agent systems, and the introduction of HospitalSim could be a useful testbed for fair multi-agent coordination. The paper is also commendable for releasing code and for reporting a wide sweep of the two penalty weights, plus comparisons with two existing fairness-aware MARL methods. However, the central gradient question about Eq. (6) is not a presentation issue: as written, the retrospective term is constant with respect to the policy parameters, so the paper's explanation of how the method works is not supported by the equations. This must be resolved before the empirical claims can be accepted.","major_comments":[{"comment":"The retrospective term α·|Ḡ_N1(τ) − Ḡ_N0(τ)| is computed from the total rewards collected in the last episode, and is therefore a constant with respect to the policy parameters θ_i. In any standard autodiff implementation of the PPO objective, its gradient with respect to θ_i is identically zero, so this term cannot by itself change the policy. This is contradicted by Table 13, where β=0 configurations such as α=0.25, β=0 (disparity 6.52 vs. PPO 6.83) are reported, and by the paper's claim that the retrospective component participates in the fairness mechanism. The supplementary mention of a 'dynamic λ' is not sufficient because λ is never defined. The authors must specify the actual objective used in code (for example, whether the penalty is folded into rewards or advantages before computing returns) and, if the α-term is genuinely used, derive its gradient or show the code path that makes it effective.","section":"Section 4, Eq. (6)"},{"comment":"In HospitalSim the fairness penalty is computed from the rewards of patients, who are not learning agents, yet Eq. (5) defines V̄_G(s_t) as the average of value estimates V_θ_i(s_t) over agents i in G. Patients have no policy parameters in the described framework, so it is undefined what V̄_N1(s_t) − V̄_N0(s_t) means for the patient groups and whether it is differentiable with respect to the three learning agents' parameters. The prospective component of the HS results therefore needs a concrete definition, such as a separate patient-value critic, before the reported trade-offs can be attributed to the stated mechanism.","section":"Section 5 (Train) and Eq. (5)"},{"comment":"Of the 25 (α, β) configurations in Table 13, only a minority produce demographic disparity below PPO's value of 6.83 (e.g., α=0.25, β=0; α=0.25, β=0.5; α=0.5, β=0.25; α=0.75, β=0.75; α=1.0, β=1.0). Moreover, the best such setting (α=0.5, β=0.25) reduces daily treated patients from 105.92 to 68.69. The abstract's statement that Fair-PPO 'achieves fairer policies than PPO' across the fairness metrics is therefore an overstatement unless it is restricted to selected configurations or accompanied by a demonstration of a systematic trend. Please report the distribution across seeds for all configurations and explicitly quantify the efficiency cost in the headline comparison.","section":"Table 13 and Section 6"},{"comment":"The retrospective penalty is computed from a single episode's total returns: a 3000-step AH game or a 12-hour, 300-patient HS day. Given the stochasticity of bush growth in AH and patient arrivals in HS, a one-episode estimate of ΔDP has high variance. If the penalty chases episodic noise rather than a stable policy-level disparity, this would explain why most HS settings do not beat PPO and why the fairest settings collapse efficiency. The paper should validate that the episodic penalty correlates with the long-run evaluation disparity, for example by showing smoothed training curves of the penalty against the evaluation-time ΔDP, or by reporting multiple training seeds.","section":"Section 5 (Train)"}],"minor_comments":[{"comment":"The paper claims counterfactual fairness as one of the evaluated fairness notions, but no counterfactual fairness results are reported; only demographic parity and conditional statistical parity tables appear. Please either add the counterfactual fairness experiments or remove the claim from the abstract.","section":"Abstract and Section 6"},{"comment":"The counterfactual system M′ is defined conceptually, but the training procedure for constructing π′ and the meaning of the counterfactual value estimates in Eq. (7) are not described. No experimental implementation of counterfactual fairness is provided.","section":"Section 4, Definition 4.2"},{"comment":"The HS description states that the simulation is event-driven and has no time steps, yet Eq. (6) and the PPO objective use per-timestep quantities indexed by t. Please clarify how episode-level penalties are attached to the timestep-level objective in the event-driven setting.","section":"Section 5 (Train/Test)"},{"comment":"The cross-references to 'Table 6 and 7' in the supplementary text appear to be outdated; the actual supplementary tables are numbered 11–14. Please correct these references.","section":"Supplementary (Additional Results)"},{"comment":"Several table headers concatenate long labels without separators, e.g., 'Fair-PPOα= 0.25, β= 0.25' and 'PatientWait Escort(Minutes)'. This makes the tables difficult to read and should be reformatted.","section":"Tables 1, 2, and 13"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the gradient of Eq. (6). If the released code shows that the α-term is actually implemented through reward shaping or another differentiable path, the authors should rewrite the objective accordingly and re-derive the mechanism; if not, the reported α-sweep results, especially the β=0 rows, are unexplained. I would ask the editor to have the authors provide the precise loss function used in the code and an ablation separating the gradient contributions of the retrospective and prospective terms."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know the main thing first: the retrospective half of the penalty in Eq. (6) is a constant with respect to the policy parameters. G_N1(τ) and G_N0(τ) are scalars from the last episode; their absolute difference has zero gradient through θ. So as written, the α-term cannot change the policy. The paper still reports α-sweeps, including a β=0 case (α=0.25, β=0) that improves disparity, which means the implementation must be doing something not in the equations — perhaps adding the penalty to rewards before computing advantages, or using the dynamic λ in a way that couples α to the critic. None of that is specified. This is a load-bearing hole, not a stylistic one.\n\nWhat is genuinely new: the prospective term β|V_N1(s_t) - V_N0(s_t)| — using critic estimates as a fairness signal — is a sensible idea I haven't seen in the cited fair-RL literature. The AH experiments, with boxplots, show a real reduction in demographic disparity (0.182/0.159 vs 0.315/0.313 for PPO). HospitalSim is a substantial new environment, and the paper honestly reports the efficiency cost, the Gini index, and a cautious ethical statement.\n\nThe soft spots are real but secondary to the gradient issue. The HS results are mixed: most (α,β) settings don't beat PPO's 6.83 disparity, the headline best (5.31) treats only 68.69 patients daily vs 105.92 for PPO, and there are no error bars or significance tests anywhere in Tables 1 and 13. Table 2 shows CSP disparity worse than PPO for two of three priority groups. Also, because Eq. (6) is called the sample-based equivalent of the evaluation metric ΔDP, the primary fairness improvement is partially built into the objective; the independent content is the prospective term and the strategy spectrum.\n\nWho is this for? Researchers working on fairness in multi-agent RL will want to engage, mainly to replicate and to fix the mechanism. The code is available, which is good. But as it stands, the formal description doesn't match the reported results. My recommendation: send to peer review, but flag the Eq. (6) gradient issue explicitly. The authors need to specify how the retrospective term enters the optimization, or remove it and rerun the α-sweep. If they can't, the paper reduces to a single-knob prospective penalty with mixed evidence.","headline":"The prospective value-based fairness penalty is a real idea, but the retrospective α-term in Eq. (6) is formally inert as written, so the paper's central mechanism needs major clarification before the results can be trusted.","tokens_in":21527,"tokens_out":3582,"would_cite":false,"duration_ms":33554,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-part fairness penalty added to PPO's objective produces fairer multi-agent policies without sacrificing competitive performance.","keywords":["fair reinforcement learning","multi-agent systems","Proximal Policy Optimization","demographic parity","counterfactual fairness","conditional statistical parity","Allelopathic Harvest","HospitalSim"],"falsifier":"Train Fair-PPO and PPO in HospitalSim on identical patient streams, then re-run the trained policies after randomly permuting the impairment labels; if the disparity reduction persists when the sensitive attribute carries no causal information, the penalty is reducing variance rather than group bias. A cheaper check is to correlate the one-episode reward gap used in Eq. (6) with the 500-episode test disparity and observe whether it is near zero.","tokens_in":20248,"feed_emoji":"⚖️","tokens_out":9739,"duration_ms":86773,"temperature":0.7,"pith_summary":"This paper claims that fairness in multi-agent reinforcement learning can be achieved by adding a penalty to the standard PPO objective rather than by designing a new fair algorithm. Fair-PPO penalises the gap between sensitive and non-sensitive groups in two places: the total rewards earned in the just-finished episode and the value estimates the critic assigns to the current state. In the Allelopathic Harvest and HospitalSim environments, the paper reports that Fair-PPO lowers demographic, conditional statistical, and counterfactual disparity relative to PPO, and that its efficiency matches the FEN and SOTO baselines. If correct, the method gives practitioners a two-knob way to move any PPO deployment along a fairness-efficiency frontier.","feed_headline":"Two-knob fairness penalty makes PPO fairer","feed_subtitle":"Fair-PPO cuts demographic disparity and lets users dial in the fairness-efficiency trade-off with two weights.","key_machinery":"The load-bearing object is the modified objective $$$L^{{\\mathrm{Fair-PPO}}$}_t(\\theta_i) = \\hat{\\mathbb{E}}_t\\left[ $L^{{\\mathrm{CLIP}}$}_t(\\theta_i) - c_1 $L^{{\\mathrm{VF}}$}_t(\\theta_i) + c_2 H[\\pi_{\\theta_i}](s_t) - \\$\\lambda$ $L^{{\\mathrm{fair}}$}_t \\right],$$ with the demographic-parity penalty $$$L^{{\\mathrm{fair-DP}}$}_t = \\$\\alpha$\\left|\\bar{G}_{N_1}(\\tau)-\\bar{G}_{N_0}(\\tau)\\right| + \\$\\beta$\\left|\\bar{V}_{N_1}(s_t)-\\bar{V}_{N_0}(s_t)\\right|.$$ The first term is retrospective, using one finished episode's group-average returns; the second is prospective, using the critic's current value estimates. The paper provides analogous penalties for counterfactual fairness and conditional statistical parity. This two-component decomposition is what makes the fairness pressure tunable: $\\alpha$ controls reaction to past disparity and $\\beta$ controls anticipation of future disparity.","core_discovery":"On its own terms, the central discovery is that a penalty term composed of a retrospective reward-disparity component and a prospective value-disparity component, inserted into the PPO clipped-surrogate objective, is sufficient to shift policies toward fairer reward distribution across groups defined by a sensitive attribute. The retrospective term compares per-episode average total rewards $\\bar{G}_{N_1}(\\tau)$ and $\\bar{G}_{N_0}(\\tau)$; the prospective term compares average critic estimates $\\bar{V}_{N_1}(s_t)$ and $\\bar{V}_{N_0}(s_t)$. The paper reports lower demographic disparity than PPO in both games; in HospitalSim the fairest tested setting cuts demographic disparity from 6.83 to 5.31 while treating 68.69 patients per day instead of 105.92, and in Allelopathic Harvest Fair-PPO matches SOTO on fairness while FEN reaches lower disparity only with much lower rewards and worse Gini. The authors conclude that fairness is purchasable at the cost of efficiency, but not at the cost of overall population equality.","pith_inferences":["A natural, untested extension is to anneal or learn $\\alpha$ and $\\beta$ during training; the HospitalSim grid suggests only a narrow band of settings improves on PPO, which an adaptive schedule might find more reliably.","Because the prospective term depends on the critic, its meaning changes as the critic improves; an implied prediction is that the optimal $\\beta$ should shrink over training as value estimates become reliable.","Nothing in the penalty formula restricts it to binary attributes, so applying the same group-disparity penalty to multi-valued or continuous sensitive attributes is a plausible next step, though the paper does not test it."],"forward_implications":["Setting $\\alpha=\\beta=0$ recovers standard PPO, so Fair-PPO is a strict generalisation of PPO and requires no change to the policy network.","Sweeping $\\alpha$ and $\\beta$ across $[0,1]$ produces a family of policies with different fairness-efficiency balances, so a user can select an operating point rather than a single fair policy.","Replacing the demographic-parity penalty with the counterfactual or conditional-statistical-parity penalty swaps the fairness notion without changing the training loop.","In HospitalSim, lower demographic disparity came with fewer daily treated patients, so the fairness gain is real but paid for in throughput."],"supporting_citations":[{"why":"Provides the PPO clipped surrogate objective that Fair-PPO modifies.","marker":"Schulman et al. 2017b"},{"why":"Supplies the idea of penalising the PPO loss for violating a constraint, which Fair-PPO adapts to fairness.","marker":"Zhang et al. 2022"},{"why":"Defines the Allelopathic Harvest environment used for the cooperative-competitive evaluation.","marker":"Leibo et al. 2019"},{"why":"FEN is a state-of-the-art fair RL baseline Fair-PPO is compared against.","marker":"Jiang and Lu 2019"},{"why":"SOTO is the other state-of-the-art fair RL baseline used to benchmark efficiency and disparity.","marker":"Zimmer et al. 2021"},{"why":"Defines counterfactual fairness, one of the three fairness metrics implemented as a penalty.","marker":"Kusner et al. 2018"},{"why":"Basis for demographic parity used in the main fairness penalty.","marker":"Kamishima et al. 2012"}],"fun_headline_variants":["Fair-PPO dials in fairness without losing equality","Retrospective and prospective penalties cut PPO bias","PPO gets fairness knob, trade-off is efficiency only","Fair-PPO: fairness at cost of efficiency, not equality","Two-part penalty steers PPO to fairer rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one episode's difference in total reward between the sensitive and non-sensitive groups, together with the critic's current value estimates, is a trustworthy signal of lasting policy unfairness rather than a sample of environment noise.","fun_headline_variants_meta":{"raw":{"variants":["Fair-PPO dials in fairness without losing equality","Retrospective and prospective penalties cut PPO bias","PPO gets fairness knob, trade-off is efficiency only","Fair-PPO: fairness at cost of efficiency, not equality","Two-part penalty steers PPO to fairer rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2525,"prompt_tokens":1000,"completion_tokens":1525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":1445}},"tokens_in":616,"tokens_out":1525,"duration_ms":10044,"temperature":1.0,"reasoning_tokens":1445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:07:21.931675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Fair-PPO and PPO in HospitalSim on identical patient streams, then re-run the trained policies after randomly permuting the impairment labels; if the disparity reduction persists when the sensitive attribute carries no causal information, the penalty is reducing variance rather than group bias. A cheaper check is to correlate the one-episode reward gap used in Eq. (6) with the 500-episode test disparity and observe whether it is near zero.","supporting_citations":[{"cited_title":"Z.; Perolat, J.; Hughes, E.; and et al","cited_arxiv_id":null,"evidence_quote":"Defines the Allelopathic Harvest environment used for the cooperative-competitive evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FEN is a state-of-the-art fair RL baseline Fair-PPO is compared against."},{"cited_title":"Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning","cited_arxiv_id":"2012.09421","evidence_quote":"SOTO is the other state-of-the-art fair RL baseline used to benchmark efficiency and disparity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Basis for demographic parity used in the main fairness penalty."}],"review_version":1}