{"id":"83bed4c1-211b-4de8-ae51-a2a6c85f0d85","arxiv_id":"2412.12326","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A suggestion-sharing MARL algorithm lets agents exchange optimized action proposals for each other, with a theoretical bound relating the surrogate objective to collective return.","lead":"This paper proposes a multi-agent reinforcement learning method where agents share action suggestions instead of rewards, values, or full policies, aiming to align individual interests with collective welfare. It provides a theoretical lower bound and tests the method on social dilemma environments, claiming performance competitive with baselines that share more private information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The lower bound in Theorem 1 is shifted by the positive term f^{pi_old}, so at pi_new=pi_old it reads eta(pi_old) >= eta(pi_old) - f^{pi_old}; maximizing Eq. 11 is thus not certified to improve collective return even in finite MDPs.","rationale":"The reader's conditional verdict and weakest_assumption identify a real scope gap: the theorem requires finite state and action spaces, while the central C. Predation environment is continuous. That concern is valid but secondary, because it limits the theorem's applicability without challenging the theorem inside its stated domain. The more load-bearing issue is that the theorem's lower bound contains a positive constant f^{pi_old} that is dropped in the optimization objective. Even in a purely finite MDP, the bound is slack at the current policy, so maximizing Eq. 11 is not proven to improve on pi_old. This matters because the paper's central claim, as summarized by the reader's strongest_claim, is that the surrogate maximization improves a lower bound on collective return. The proof of Theorem 1 appears algebraically reasonable, and the f term does not affect the argmax of the surrogate, so the algorithm can still be viewed as a heuristic. The empirical results provide independent support and should be credited. For those reasons, the correct verdict remains CONDITIONAL rather than ACCEPT or REJECT; the reader's existing conditional verdict already accommodates this concern, so no change is needed.","tokens_in":19921,"tokens_out":10980,"duration_ms":106425,"concrete_test":"Analytical check: evaluate Eq. 10 at pi_new = pi_old and pi_tilde_i^new = pi_old for all i. If the right-hand side is eta(pi_old) - f^{pi_old} with f^{pi_old} > 0, the bound is not anchored at the incumbent policy. Computational follow-up: on a finite two-agent version of the C. Predation environment (discretize X=[0,30] into 30 bins, A={-1,+1}), solve the constrained surrogate in Eq. 11 over a grid of delta and delta-prime and compute the best achievable right-hand side of Eq. 10. If no feasible point attains a lower bound at least eta(pi_old), the theorem provides no improvement guarantee, and the theoretical support should be weakened from 'guaranteed improvement' to 'surrogate heuristic.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that maximizing the suggestion-based surrogate under the stated constraints improves a lower bound on collective return. Theorem 1 (Eq. 10) gives eta(pi_new) >= eta(pi_old) + zeta^{pi_old}(Pi_tilde_new) - C * sum_i D_KL^max(pi_ii^old || pi_ii^new) - f^{pi_old} - sum_i (1/2) max_{s,a}|A_i^{pi_old}| * sum_{s,a}(pi_tilde_i^new(a|s) - pi_new(a|s))^2. The term f^{pi_old}, defined in Eq. 8, is positive in any nontrivial finite MDP with nonzero advantages. Evaluate the bound at the current policy with exact suggestions, pi_new = pi_old and pi_tilde_i^new = pi_old for all i. Then zeta = 0, both penalty terms vanish, and Eq. 10 reduces to eta(pi_old) >= eta(pi_old) - f^{pi_old}, a slack inequality. Thus the lower bound is not anchored at the incumbent return; it starts below eta(pi_old) by an amount f^{pi_old}. Section 4.2 discards f^{pi_old} because it is 'bounded,' but boundedness is irrelevant: the negative constant weakens the guarantee. The constraints in Eq. 12 limit KL divergence and pairwise policy discrepancies, but they impose no condition such as zeta - C*sum KL - discrepancy >= f^{pi_old}. Therefore the theorem does not establish that the proposed updates improve on pi_old. Maximizing a lower bound that drops the constant is still a reasonable heuristic, but the paper's stated claim that sharing suggestions aligns individual policies with the collective objective is not formally supported by Theorem 1. This concern applies inside the theorem's finite-state assumptions, so it is more fundamental than the continuous-state gap in C. Predation, although that gap is also real: the sums over s and |A| in Eq. 8 are undefined for the continuous state space X=[0,30] used in Section 5.1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Suggestion Sharing (SS), a multi-agent reinforcement learning method for collective welfare under individual rewards. Each agent learns a suggesting joint policy composed of its own policy and action suggestions for other agents, and agents exchange only these action distributions rather than rewards, values, or full policies. The authors derive a TRPO-style lower bound on the collective return (Theorem 1) in terms of the sum of individual advantages evaluated under the suggesting joint policies, minus KL divergence penalties, a joint-action discrepancy penalty, and a constant f^{π_old}. From this bound they formulate a constrained surrogate objective (Eqs. 11--12) and a practical PPO-style objective with clipping and indicator penalties (Eq. 16). Experiments on Cleanup, Harvest, Cooperative Navigation, and Cooperative Predation show that SS performs competitively with baselines that share values, policies, or use intrinsic rewards.","tokens_in":20414,"tokens_out":13229,"duration_ms":120705,"significance":"If fully established, the theoretical claim would be significant: SS offers a privacy-motivated communication protocol that exchanges only action suggestions, and the empirical results suggest it can resolve sequential social dilemmas without value, reward, or policy sharing. The paper includes detailed proofs in the appendix and a clean ablation (ρ = 0) showing that the suggestion-based penalty is essential for performance. However, the theoretical support for the algorithm is currently incomplete: the lower bound is shifted by a negative constant, the practical objective is heuristic rather than derived from the bound, and the finite-state theorem does not cover the continuous-state C. Predation experiments. The empirical contribution is useful, but the claim that the algorithm is supported by the lower-bound analysis needs substantial reframing.","major_comments":[{"comment":"The lower bound in Theorem 1 contains the negative constant −f^{π_old}. Evaluating at the incumbent policy with exact suggestions, π_new = π_old and tildeπ_i_new = π_old for all i, reduces Eq. (10) to η(π_old) ≥ η(π_old) − f^{π_old}, so the bound is slack by the positive amount f^{π_old}. Since dropping a constant does not change the argmax, maximizing Eq. (11) is not certified to produce a policy with η(π_new) ≥ η(π_old); the theorem only relates η(π_new) to a shifted quantity. The boundedness argument in Section 4.2 is not sufficient: a bounded negative constant still weakens the guarantee. The authors should either add a condition such as ζ^{π_old}(tildeΠ_new) − C·Σ_i D_KL^max(π_ii_old||π_ii_new) − discrepancy ≥ f^{π_old}, or explicitly state that the surrogate is a heuristic motivated by the lower bound rather than a certified monotone improvement step. This is load-bearing because the abstract and introduction claim that SS is supported by a lower-bound analysis.","section":"4.2, Eqs. (10)--(11)"},{"comment":"The practical objective in Eq. (16) is not shown to follow from the constrained problem in Eq. (12) or from Theorem 1. It replaces the hard constraints of Eq. (12) with indicator penalties whose activation sets X_ij and X_ii are defined by comparing signed quantities in Eq. (14); the accompanying text explains only the case A_i > 0, and for negative advantages the inequality condition reverses. More importantly, no argument establishes that optimizing Eq. (16) decreases the joint-action discrepancy term in Theorem 1 or satisfies constraints (b) and (c) of Eq. (12). The penalty coefficient ρ is a free parameter that is tuned separately per environment (Table 3). Consequently, the empirical performance of SS is not formally explained by Theorem 1; the paper should either prove a connection between Eq. (16) and the theoretical surrogate, or present Eq. (16) as a heuristic algorithm motivated by, but not guaranteed by, the theory.","section":"4.3, Eq. (16)"},{"comment":"Theorem 1, Lemma 1, and Lemma 2 are proved for finite state and action spaces: the proofs sum over s ∈ S, use |A| (the size of the joint action space), and use ∥d^{π'}∥_2^2 = Σ_s (d^{π'}(s))^2. In C. Predation (Section 5.1), the state space is continuous, X = [0,30], so f^{π_old} and the discrepancy sums over states are not defined. Since C. Predation is the main sequential social dilemma environment used to demonstrate the method (Figs. 1, 3--5), the stated theorem does not cover a central part of the empirical evaluation. The paper should either state the finite-state assumption explicitly and mark C. Predation as outside the theoretical scope, or provide a continuous-state extension.","section":"5.1 and Theorem 1"},{"comment":"The constraints in Eq. (12) bound pairwise marginal discrepancies between π_ij and π_jj (constraint b) and between π_ii and π_ji (constraint c), while the penalty term in Theorem 1 is Σ_{s,a} (tildeπ_i_new(a|s) − π_new(a|s))^2 over joint actions. The paper does not provide an inequality relating these quantities. For product policies, the joint-action L2 distance is not bounded by the sum of marginal L2 distances in the form used here without additional constants depending on N and |A|. Thus, satisfying Eq. (12) does not by itself imply that the last penalty term in Eq. (10) is small, and the claimed decomposition of the surrogate objective is incomplete.","section":"4.2, Eqs. (11)--(12)"}],"minor_comments":[{"comment":"In the proof of Theorem 1, Eq. (37) writes max_s d^{π_old}(s)^2 inside the sum over states and actions; this should be d^{π_old}(s)^2 for the subsequent replacement by f^{π_old} via Eq. (31) to be valid. Also, the sentence 'Using Theorem 2' in the same proof should read 'Using Lemma 2'.","section":"Appendix A.1.3, Eq. (37)"},{"comment":"The relationship between total variation and KL divergence is written as (D_TV(p||q))^2 ≤ D_KL(q||q); this should be D_KL(p||q).","section":"Appendix A.1.1, proof of Lemma 1"},{"comment":"The set X_ii in Eq. (14) depends on π_ji, which is different for each neighbor j, but in Eq. (16) it is written without a neighbor index inside the sum over j ∈ N_i; the notation should be clarified, for example by writing X_ii^{(j)} or by defining the set per neighbor.","section":"Eqs. (14) and (16)"},{"comment":"The paper reports that 5 runs with different seeds were conducted, but the training curves and final results do not show error bars or confidence intervals; reporting variability would strengthen the claim that SS performs competitively with the baselines.","section":"Section 5.3, Figs. 1--2"},{"comment":"Several references are incomplete, including 'Siedler and Alpha' (missing title and venue) and 'Macy and Flache' (missing year and complete venue); the reference for 'Huang and Zhou' also lacks a year. Please complete the bibliography.","section":"References"},{"comment":"The penalty coefficient ρ varies by orders of magnitude across environments (1e3 in Cleanup, 0.1 in Harvest and C. Predation, 1 in C. Navigation). Since the ablation in Fig. 4 indicates that this term is important, the selection procedure for ρ should be described, or a sensitivity analysis should be provided.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The suggestion-sharing idea is novel and the empirical results are promising, but the theoretical framing overstates what is proved. In my assessment the three core problems—the unanchored lower bound due to −f^{π_old}, the missing derivation of Eq. (16) from Eq. (12), and the finite-state/continuous-state mismatch with C. Predation—are addressable by rewriting the theoretical claims and adding the missing analysis, so I recommend major revision rather than rejection. The authors should also be encouraged to release code to allow verification of the experimental comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The suggestion-sharing idea is genuinely new: each agent learns and shares action distributions that would help other agents, rather than sharing rewards, values, or full policies. The lower-bound analysis is a real extension of TRPO to individual-reward multi-agent settings, and the experiments span four social dilemmas with sensible baselines. The paper deserves a serious referee, but the theory should be tightened before publication.\n\nThe main soft spot is Theorem 1. The lower bound includes the negative constant f^{pi_old}, which makes the bound slack at the current policy: plugging pi_new = pi_old gives eta(pi_old) >= eta(pi_old) - f, not tightness. The surrogate in Eq. 11 maximizes the advantage term while constraining the penalty terms, but nothing in those constraints guarantees that the surrogate gain outweighs f. So the paper's claim that optimizing the lower bound provably improves collective return is not actually supported. The authors drop f as 'bounded,' but boundedness doesn't restore the monotonicity argument. This is a genuine gap, though its practical severity is unclear because f contains ||d^{pi_old}||_2^2, which is often small in large state spaces.\n\nA secondary issue is the finite-state assumption. The sums and |A| in Lemma 2 and Theorem 1 require finite state and action spaces, yet C. Predation has a continuous state space. That's a real mismatch, but it's standard practice to apply discrete-action theory to function approximation, so I'd treat it as a coverage gap rather than a fatal flaw.\n\nOne point where the reader's concern may be off: the indicator penalties in Eq. 14 actually handle negative advantages correctly. The inequality pi_ij/pi_jj * A_i >= A_i flips to pi_ij/pi_jj <= 1 when A_i is negative, which is exactly the condition that the gradient is increasing the discrepancy. So the sign logic is fine.\n\nOverall: the mechanism is novel and the experiments are credible. The theory needs revision—either add a condition that the surrogate gain overcomes f, or explicitly state that the bound is not tight and the optimization is a heuristic guided by the bound. I would send this to peer review, but I'd ask the authors to fix the theoretical framing before acceptance.","headline":"New suggestion-sharing mechanism with credible experiments, but the theoretical guarantee in Theorem 1 is weaker than claimed because of the dropped f^{pi_old} constant.","tokens_in":20905,"tokens_out":4634,"would_cite":true,"duration_ms":44625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Suggestion Sharing, a multi-agent reinforcement learning method where agents exchange only action suggestions, and proves a lower bound showing that maximizing the suggestion-based surrogate improves the collective…","keywords":["multi-agent reinforcement learning","collective welfare","social dilemmas","suggestion sharing","action suggestions","trust region policy optimization","decentralized cooperation","privacy-preserving MARL"],"falsifier":"On a finite-state social dilemma (or a discretized version of Cooperative Predation), compute the right-hand side of Theorem 1 from logged policies and rewards and compare it with the realized collective return; the theorem is falsified if the realized return is ever below the bound.","tokens_in":19726,"feed_emoji":"🤝","tokens_out":8230,"duration_ms":62722,"temperature":0.7,"pith_summary":"This paper proposes Suggestion Sharing (SS), a multi-agent reinforcement learning method for settings where each agent pursues its own reward but the goal is collective welfare, as in social dilemmas. It claims that agents can align their policies with the group objective by exchanging action suggestions—proposals for how others should act—without sharing rewards, value functions, or full policies. The argument rests on a lower bound (Theorem 1) that ties the collective return of the updated joint policy to a suggestion-based surrogate objective, minus penalties for how far each agent's own policy and suggestions stray from the reference policies. A practical algorithm approximates this bound with a clipped, PPO-style objective and discrepancy penalties, and experiments across Cleanup, Harvest, Cooperative Navigation, and Cooperative Predation show it matches or exceeds baselines that share values, policies, or intrinsic rewards. If correct, SS offers a privacy-conscious route to cooperation when rewards and policies are sensitive.","feed_headline":"Action suggestions alone align agents with collective welfare","feed_subtitle":"Exchanging only action proposals keeps rewards and policies private while matching value-sharing baselines.","key_machinery":"The load-bearing object is the suggesting joint policy, $\\tilde{\\pi}_i(a|s)=\\prod_j \\pi_{ij}(a_j|s)$, where $\\pi_{ii}$ is agent $i$'s own policy and $\\pi_{ij}$ is the action suggestion agent $i$ holds for agent $j$. The theoretical engine is Theorem 1, which lower-bounds $\\eta(\\pi^{\\mathrm{new}})$ by $\\eta(\\pi^{\\mathrm{old}}) + \\zeta^{\\pi^{\\mathrm{old}}}(\\tilde{\\Pi}^{\\mathrm{new}}) - C\\sum_i D_{\\mathrm{KL}}^{\\max}(\\pi_{ii}^{\\mathrm{old}}\\|\\pi_{ii}^{\\mathrm{new}}) - f^{\\pi^{\\mathrm{old}}} - \\sum_i \\frac{1}{2}\\max_{s,a}|A_i^{\\pi^{\\mathrm{old}}}| \\sum_{s,a}(\\tilde{\\pi}_i^{\\mathrm{new}}(a|s)-\\pi^{\\mathrm{new}}(a|s))^2$; the first two terms form the surrogate objective, and the penalties become constraints in the per-agent optimization. The practical algorithm implements the bound through a dual clipped objective (Eq. 16) that combines a PPO-style clipped ratio for the agent's own policy with penalty terms activated exactly when a gradient step would widen the discrepancy between suggestions and actual policies.","core_discovery":"On the paper's own terms, the central discovery is that the gap between individual and collective objectives in MARL can be controlled by the discrepancy between the policies agents actually follow and the suggestions they exchange. Specifically, Theorem 1 states that the collective return of a new joint policy is at least the return of the old policy plus the expected sum of individual advantages evaluated under each agent's suggesting joint policy $\\tilde{\\pi}_i$, minus a KL-divergence penalty on each agent's own policy change and minus a squared-discrepancy penalty between the suggesting joint policies and the true joint policy. Because the objective decomposes per agent, each agent can maximize its own advantage subject to constraints that keep its suggestions close to others' actual policies and its own policy close to others' suggestions. The paper then turns this bound into the Suggestion Sharing algorithm, in which agents share only action distributions and suggestions, and shows empirically that this suffices to reach cooperation in social dilemmas.","pith_inferences":["Beyond the paper's own claims, a testable extension is to verify whether the stated bound holds in continuous-state dilemmas: Theorem 1 sums over states and actions, while the Cooperative Predation environment uses a continuous state space, so the empirical result there is not covered by the theorem.","The suggestion-sharing mechanism can be read as a soft contracting protocol: because each agent optimizes suggestions that would benefit itself, the exchanged suggestions carry implicit information about the sender's interests; quantifying that leakage (for instance, via mutual information between suggestions and rewards) would sharpen the paper's qualitative privacy claim.","The assumption that agents truthfully share suggestions is an untested vulnerability; injecting one deceptive agent that sends suggestions steering others toward its own benefit is a direct stress test for whether collective welfare survives without trust mechanisms.","The KL-decomposition property for product policies is generic, so the lower-bound proof may transfer to other trust-region MARL settings that keep the same per-agent decomposition, such as team-reward or mixed-motive games."],"forward_implications":["If Theorem 1 holds, maximizing the suggestion-based surrogate under the stated constraints improves a lower bound on collective return, so agents can cooperate without a shared reward or value signal.","Because only action distributions and suggestions are exchanged, SS is designed to reveal less private information than methods that share rewards, value functions, or policy parameters, making it relevant where such sharing is infeasible.","The constraint penalties in Eq. 12 are what keep the suggestions informative; the ablation with $\\rho=0$ shows that without them performance drops, confirming the mechanism.","In sequential social dilemmas like Cooperative Predation, SS converges to joint cooperation with a C-C rate near 1 despite individual incentives to defect.","Because the objective decomposes into per-agent terms, SS can run with only local communication with neighbours, and the scalability study shows that reducing communication frequency or the number of neighbours still yields competitive performance."],"supporting_citations":[{"why":"Supplies the trust-region lower-bound machinery that Lemma 1 adapts to multi-agent settings with individual advantages.","marker":"[Schulman et al., 2015]"},{"why":"Provides the PPO-style clipping used in the SS update to keep each agent's own policy close to its previous policy.","marker":"[Schulman et al., 2017]"},{"why":"Provides generalized advantage estimation, used to estimate the individual advantage terms in the SS objective.","marker":"[Schulman et al., 2016]"},{"why":"Supplies the product-policy KL property used in the proof of Theorem 1 to split the joint-policy KL divergence per agent.","marker":"[Su and Lu, 2022]"},{"why":"Supplies the Cleanup and Harvest social-dilemma environments used to evaluate SS.","marker":"[Christoffersen et al., 2023]"},{"why":"Provides the Cooperative Navigation environment, the networked-MMDP formalism, and the VPS baseline.","marker":"[Zhang et al., 2018b]"},{"why":"Provides the Value Sharing baseline that SS is compared against.","marker":"[Du et al., 2022]"},{"why":"Provides the Policy Sharing baseline that SS is compared against.","marker":"[Zhang and Zavlanos, 2019]"},{"why":"Provides the Intrinsic Moral Rewards baseline that SS is compared against.","marker":"[Tennant et al., 2023]"}],"fun_headline_variants":["Suggestion sharing alone bridges the individual-collective gap in MARL","Sharing only suggestions aligns MARL agents with collective welfare","Action suggestion exchange matches value-sharing cooperation in MARL","Suggestion sharing aligns agents with collective goals without value sharing","Exchanging suggestions, not rewards, yields collective welfare in MARL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the stated lower bound actually applies, which requires finite state and action spaces and suggestions that stay close to others' true policies; if either condition fails, the theoretical justification for SS collapses.","fun_headline_variants_meta":{"raw":{"variants":["Suggestion sharing alone bridges the individual-collective gap in MARL","Sharing only suggestions aligns MARL agents with collective welfare","Action suggestion exchange matches value-sharing cooperation in MARL","Suggestion sharing aligns agents with collective goals without value sharing","Exchanging suggestions, not rewards, yields collective welfare in MARL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4224,"prompt_tokens":913,"completion_tokens":3311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":3226}},"tokens_in":529,"tokens_out":3311,"duration_ms":21187,"temperature":1.0,"reasoning_tokens":3226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:13:05.679111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a finite-state social dilemma (or a discretized version of Cooperative Predation), compute the right-hand side of Theorem 1 from logged policies and rewards and compare it with the realized collective return; the theorem is falsified if the realized return is ever below the bound.","supporting_citations":[{"cited_title":"Trust region policy optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the trust-region lower-bound machinery that Lemma 1 adapts to multi-agent settings with individual advantages."},{"cited_title":"Jordan, and Pieter Abbeel","cited_arxiv_id":null,"evidence_quote":"Provides generalized advantage estimation, used to estimate the individual advantage terms in the SS objective."},{"cited_title":"Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning","cited_arxiv_id":"2301.08491","evidence_quote":"Provides the Intrinsic Moral Rewards baseline that SS is compared against."}],"review_version":1}