{"id":"567bbf6f-6536-4cdd-9776-2d74673cd339","arxiv_id":"2501.01266","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"In a bespoke multi-agent exploration task, a reward that gives agents a share of their peers' rewards proportional to their influence raises team return, though a simpler influence-only variant explores more state space.","lead":"This paper introduces PIMAEX, a peer-incentivized reward for multi-agent reinforcement learning that rewards agents for influencing teammates toward novel states, alongside a communication-based training algorithm. The authors test it on a new Consume/Explore environment and report higher team returns than vanilla PPO and PPO with random network distillation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim that the novel β term improves return is confounded: Table 5 changes reward term together with communication loss weight and influence measure, so β's advantage may be due to hyperparameters rather than the β reward.","rationale":"The paper honestly describes its limitations and reports per-seed statistics, and the in-environment result for the full PIMAEX-Communication package is plausibly real. However, the paper's stated main contribution is the novel β term, and the empirical support for that term is confounded. In the single-term comparison, PIMAEX β and PIMAEX α differ not only in which term is active but also in communication loss weight (1.0 vs 0.0758) and influence measure (PMI vs KLD), both of which materially affect the communication policy's learning. Without the missing ablation cells, the highest-return result cannot be attributed to the β term. The separate comparison against PPO+RND also lacks a communication-channel control, so the reward's marginal contribution is not isolated. These are internal-validity issues, not just generality concerns. The right fix is a small set of additional experiments; if they confirm β's advantage under matched hyperparameters, the claim stands. Thus the reader's CONDITIONAL verdict remains appropriate, with the added condition that the authors run the matched ablation and a communication-only baseline.","tokens_in":12061,"tokens_out":9950,"duration_ms":93005,"concrete_test":"Run a 2x2 ablation in Consume/Explore with the same 3 seeds: (1) α term with PMI and communication loss weight 1.0, (2) β term with KLD and communication loss weight 0.0758, (3) β with KLD and weight 1.0, and (4) α with PMI and weight 0.0758; additionally include a PPO+RND baseline equipped with a communication channel but no PIMAEX reward (communication-only). If β's joint-return advantage over α disappears when influence measure and communication loss weight are matched, the β-term claim is unsupported; if β remains best in all matched comparisons, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the β term in Eq. 4, and Section 6 claims PIMAEX β achieves the highest overall return. However, the 'single-term' comparison is not a controlled ablation. Table 5 sets PIMAEX β to β=1.0, α=0, γ=0 with communication loss weight 1.0 and PMI as the influence measure, while PIMAEX α is α=1.0, β=0, γ=0 with communication loss weight 0.0758 and KLD. Because two hyperparameters change together with the reward term, the β advantage could equally be explained by the 13.2x larger communication loss weight (1.0 vs 0.0758) or by the choice of PMI over KLD. The missing cells—α with PMI and loss weight 1.0, β with KLD and loss weight 0.0758—are never reported, so the paper cannot attribute the observed superiority to the β term. A second, related confound is that the PPO+RND baseline lacks the communication channel entirely, so the headline 'PIMAEX reward with PIMAEX-Communication outperforms baselines' also conflates the reward with the presence of communication. The β-vs-α comparison is the sharper issue because all PIMAEX variants share the communication channel.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PIMAEX, a peer-incentivized multi-agent exploration reward, and PIMAEX-Communication, a training algorithm with a discrete communication channel and counterfactual influence estimation. The reward is a weighted sum of three influence terms: alpha (policy influence), beta (policy influence multiplied by the peer's reward, claimed as the novel contribution), and gamma (value influence). The method is evaluated on a purpose-built partially observable environment, Consume/Explore, with PPO and PPO+RND baselines. Single-term variants PIMAEX alpha, beta, and gamma are compared. The reported results show that the best PIMAEX beta configuration achieves the highest joint episode return and lower variance than baselines in this environment. The conclusion is appropriately narrowed to overall return in Consume/Explore compared with baselines without social influence, but the attribution of this advantage to the beta term is not established by the reported experiments.","tokens_in":12331,"tokens_out":4418,"duration_ms":48874,"significance":"If the beta-term advantage were cleanly established, this would be a modest but useful contribution to multi-agent exploration, since the beta term is simple and the generalized reward formulation is a convenient unification of prior influence-based rewards. The paper is clearly written, the environment is described transparently with parameters and hyperparameters, and the authors explicitly acknowledge the main scope limitations. However, the paper provides no code or other reproducibility artifact, the evaluation is on a single self-designed environment, and the experimental design does not isolate the beta term. The main value of the paper at this stage is as a proposal of a reward form and an environment, not as a validated mechanism.","major_comments":[{"comment":"The comparison between PIMAEX alpha and PIMAEX beta is not a controlled ablation, and this is load-bearing for the paper's main claim that the novel beta term improves return. In Table 5, PIMAEX beta uses communication loss weight 1.0 and PMI as the influence measure, while PIMAEX alpha uses communication loss weight 0.0758 and KLD. Thus two hyperparameters change together with the reward term, and the reported advantage of beta over alpha could be due to the 13.2x larger communication loss weight, the choice of PMI, or their interaction. The missing cells—alpha with PMI and loss weight 1.0, and beta with KLD and loss weight 0.0758—are needed to attribute the improvement to the beta term. As it stands, Sections 6 and 7 overstate the evidence for the beta contribution.","section":"Section 6, Table 5"},{"comment":"The paper reports only the best-performing hyperparameter configuration for each method, selected on the same evaluation environment, and provides no statistical significance tests. With three seeds per model and no per-seed confidence intervals or hypothesis tests, statements such as \"PIMAEX beta exhibits significantly less standard deviation than other methods\" (Section 6) are not supported. Moreover, Figures 1 and 3 are described as the \"best training run of each agent category\" while the methodology says results were averaged over three seeds; this ambiguity affects which quantity is being plotted. The authors should report all searched configurations or a clear selection procedure, use a validation/test split if hyperparameters are tuned, and provide per-seed results or significance tests for the central return comparison.","section":"Section 5.2 and Sections 6-7"},{"comment":"The comparison against PPO+RND conflates the PIMAEX reward with the presence of the communication channel. PPO+RND has no communication channel, so the headline comparison \"PIMAEX reward with PIMAEX-Communication outperforms those that do not\" does not isolate the reward function's contribution. A more appropriate control would be a PPO+RND agent with the same communication channel and counterfactual machinery but without the social influence reward, or a PIMAEX-Communication variant with the reward terms set to zero. This issue is less sharp than the alpha-versus-beta confound because all PIMAEX variants share the communication channel, but it affects the abstract's claim that agents using the PIMAEX reward outperform those that do not.","section":"Section 4 and Section 6"},{"comment":"The paper's own limitation statement acknowledges that the evaluation was limited to a single task with small state and action spaces and short training. This is not by itself an error, but the central claim of the paper is empirical, so the lack of any second environment or scalability evidence means the general conclusion is necessarily tentative. The abstract should be aligned with the conclusion's narrow phrasing: the demonstrated results are for Consume/Explore, not for multi-agent exploration generally. At minimum, the title and abstract should not imply broader generality than the single environment supports.","section":"Section 7"}],"minor_comments":[{"comment":"The notation PI^alpha and PI^beta is confusing because alpha and beta are also the weights in the same equation. The superscripts appear to denote policy influence variants, but the text does not define a distinct PI^alpha versus PI^beta; please clarify by renaming the influence measures or the weights.","section":"Equation (4)"},{"comment":"The symbols beta_env and beta_int in Equation (5) reuse the Greek letter beta, which already denotes the second reward term in Equation (4). This is a readability issue; consider using different coefficient names for the intrinsic/extrinsic mixing weights.","section":"Equations (5) and (6)"},{"comment":"The hyperparameter search for PPO+RND and PIMAEX-Communication lists the searched values in Tables 3 and 4 but does not specify the search procedure, the number of trials, or the criterion for selecting the \"best\" configuration. Reporting this would help assess the risk of overfitting the reported result to the evaluation environment.","section":"Section 5.2"},{"comment":"The abstract states that \"agents using the PIMAEX reward with PIMAEX-Communication outperform those that do not\" without noting that this holds for a single environment and for best-tuned configurations. The conclusion is more careful; please align the abstract with the qualified claim.","section":"Abstract"},{"comment":"Figure 4(a) reports per-episode exploration state space coverage, but the text notes that agents cover only a small portion of the agent state space within one episode and therefore reports local state space coverage in Figure 4(b). It would help to state explicitly why exploration state space coverage remains meaningful within an episode despite the small agent-state coverage.","section":"Figure 4(a)"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations and the environment is described transparently, but the central novelty—the beta term—is not isolated in the experiments. A controlled ablation with the missing cells in Table 5, per-seed statistics, and an explicit statement about best-of-search selection would be necessary before I could recommend acceptance. The paper fits the journal's scope; the issue is experimental support, not novelty theft or framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an incremental but competently made contribution. The generalized social-influence reward in Eq. (4) is a clean unification of Jaques et al. and Wang et al., and the beta term forcing influence times peer reward is new, as far as I can tell. The Consume/Explore environment is small but thoughtfully built to combine deceptive reward, credit assignment, and coordination. The authors also do something rare: the conclusion states the limitations without spin and even reports that alpha, not beta, produced the most exploratory behavior. That honesty buys real goodwill.\n\nThe experimental section is the weak part, and the stress-test note is correct. Table 5 is not an ablation. PIMAEX beta differs from PIMAEX alpha in both the reward term and two hyperparameters: communication loss weight (1.0 vs 0.0758) and influence measure (PMI vs KLD). So the claim that the beta term achieves the highest return cannot be attributed to beta alone; it could be the larger communication loss weight or the choice of PMI. The missing cells (alpha with PMI and weight 1.0, beta with KLD and weight 0.0758) are exactly the cells that would settle it. The comparison against PPO+RND also conflates the reward with the communication channel, since that baseline lacks communication. That is less serious because all PIMAEX variants share the channel, but it still qualifies the headline.\n\nOther soft spots: three seeds, no significance tests, no released code or data, and one self-designed environment with a small state/action space and short training. Those are addressable, and the authors already list most of them. If the central claim were just that the PIMAEX formulation can work in this toy setting, the evidence would be enough. It is the beta-specific claim that exceeds the data.\n\nWho should read it: anyone working on influence-based rewards or multi-agent exploration might want the unified formulation and the environment. I would not cite it as evidence that beta improves return, but I would cite Eq. (4) and Consume/Explore as related work. A serious editor should send this to review rather than desk-reject: the idea is clean, the writeup is careful, the related work is fair, and the confounds are fixable with a modest set of additional experiments. I would ask for the controlled ablation, a code release, and ideally a second environment before accepting.","headline":"A clean, incremental MARL contribution whose headline beta comparison is confounded by hyperparameter choices; the generalized reward and environment are citable, but the specific beta claim needs a controlled ablation.","tokens_in":12884,"tokens_out":3239,"would_cite":true,"duration_ms":33254,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewarding influence on peers improves multi-agent exploration","keywords":["multi-agent reinforcement learning","exploration","intrinsic curiosity","social influence","peer incentivization","random network distillation","sequential social dilemma","communication"],"falsifier":"Train PIMAEX-Communication ($\\beta$) and PPO+RND on a second, independently designed partially observable coordination task with sparse rewards, using the same hyperparameter search budget and seeds; if $\\beta$ does not exceed RND in mean team return or variance reduction, the central empirical claim fails to generalize beyond its single environment.","tokens_in":11866,"feed_emoji":"🤝","tokens_out":10230,"duration_ms":76501,"temperature":0.7,"pith_summary":"This paper proposes a reward-shaping scheme that turns one agent's discovery of novel states into a reward for the agents that influenced it. The idea is a generalized social-influence reward with three weighted terms: policy influence, value influence, and a newly introduced term that multiplies an agent's influence on a peer by that peer's reward. The authors instantiate the scheme as PIMAEX, pair it with a communication-based actor-critic algorithm called PIMAEX-Communication, and evaluate it in a custom partially observable environment with deceptive rewards. They report that PIMAEX agents outperform both vanilla PPO and PPO augmented with random network distillation, with the $\\beta$ variant achieving the highest team return and the lowest variance across seeds.","feed_headline":"Rewarding influence on peers improves multi-agent exploration","feed_subtitle":"Multiplies influence by the peer's reward, boosting team return and stability in the Consume/Explore task.","key_machinery":"The load-bearing object is the generalized multi-agent social-influence reward function of Eq. 4, $r_j = \\sum_{k \\neq j} [\\alpha \\cdot PI^\\alpha_{j \\to k} + \\beta \\cdot PI^\\beta_{j \\to k} \\cdot r^w_k + \\gamma \\cdot VI^w_{j \\to k}]$. PIMAEX is the special case where $r^w_k$ and $VI^w_{j \\to k}$ are weighted sums of extrinsic and intrinsic (RND-based) rewards and values. The training side is PIMAEX-Communication, which adds a discrete communication head to each agent, samples counterfactual messages to build marginal policies and values, and computes influence on the actor side while intrinsic and PIMAEX rewards are computed on the learner side. The mechanism works by turning the communication channel into a channel of influence that the reward explicitly prices.","core_discovery":"The paper's central claim is that multi-agent exploration can be improved by giving each agent a reward that depends on its causal influence on the exploration of others. Concretely, agent $j$ receives $r_j = \\sum_{k \\neq j} [\\alpha \\cdot PI^\\alpha_{j \\to k} + \\beta \\cdot PI^\\beta_{j \\to k} \\cdot r^w_k + \\gamma \\cdot VI^w_{j \\to k}]$, where the $\\alpha$ term measures policy influence (the divergence or pointwise mutual information between a peer's informed and counterfactually marginalized policy), the $\\gamma$ term measures value influence (the difference between a peer's informed and marginalized value), and the $\\beta$ term—the paper's new contribution—multiplies influence by the peer's weighted reward, letting $j$ share in the outcome of actions it caused. The authors claim this $\\beta$ term is what lifts overall return in their Consume/Explore task, and they further observe that the $\\alpha$-only agent explores most while $\\beta$ yields the most stable policies. The empirical support is an internal comparison between best-tuned models of each reward variant.","pith_inferences":["A direct testable extension is to re-run the same comparison on a second partially observable coordination task with a larger state space; if the $\\beta$ advantage disappears, the claim is limited to the Consume/Explore setting.","The low-variance property of $\\beta$ could be investigated mechanically by logging per-step PIMAEX rewards and advantage estimates during training to see whether the $\\beta$ term compresses the spread of returns across agents.","Since the generalized reward separates influence measurement from the reward stream, one could substitute learned or learned-models-based influence measures and preserve the framework; the paper does not test this.","The pairwise specialization pattern suggests a testable prediction: in tasks with complementary roles, peer incentives should amplify role differentiation, measurable with a role-divergence metric."],"forward_implications":["Any actor-critic MARL algorithm can be augmented with PIMAEX by adding a communication policy, counterfactual message sampling, and the three-term reward, without changing the underlying policy updates.","The $\\beta$ term's low variance suggests peer-incentivized rewards can act as a variance-reduction mechanism, which the paper ties to social influence's known stabilizing effect on policy gradients.","The Consume/Explore environment provides a tunable sequential social dilemma (credit-assignment difficulty, coordination threshold, resource abundance) for future MARL exploration studies.","The $\\alpha$-only variant's strong exploration indicates that policy influence combined with individual intrinsic curiosity may be sufficient to drive exploration, even without sharing intrinsic returns.","PIMAEX $\\beta$ agents specialize into explorer and consumer roles in pairs, suggesting the reward induces division of labor within the team."],"supporting_citations":[{"why":"Supplies the $\\alpha$ policy-influence term, the discrete communication channel, and the counterfactual marginalization used in PIMAEX-Communication.","marker":"(Jaques et al., 2018)"},{"why":"Supplies the $\\gamma$ value-influence term (Value of Interaction) that the generalized reward subsumes.","marker":"(Wang et al., 2019)"},{"why":"Provides random network distillation, the intrinsic curiosity baseline and the RND mechanism used for intrinsic rewards in PIMAEX.","marker":"(Burda et al., 2018)"},{"why":"The underlying PPO actor-critic algorithm on which PIMAEX-Communication is built.","marker":"(Schulman et al., 2017)"},{"why":"The reinforcement learning framework whose PPO implementation this work extends.","marker":"(Hoffman et al., 2020)"},{"why":"The sequential social dilemma framework that the Consume/Explore environment instantiates.","marker":"(Leibo et al., 2017)"}],"fun_headline_variants":["Peer incentivization improves multi-agent exploration","Influence multiplied by peer reward boosts exploration","Multi-agent exploration via causal peer influence","Reward sharing elevates team exploration in MARL","PIMAEX: spur exploration by incentivizing peer influence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The single Consume/Explore environment, designed by the authors, is a valid and representative testbed for multi-agent exploration, and the separate hyperparameter search run for each method does not inadvertently favor PIMAEX.","fun_headline_variants_meta":{"raw":{"variants":["Peer incentivization improves multi-agent exploration","Influence multiplied by peer reward boosts exploration","Multi-agent exploration via causal peer influence","Reward sharing elevates team exploration in MARL","PIMAEX: spur exploration by incentivizing peer influence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1548,"prompt_tokens":970,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":509}},"tokens_in":586,"tokens_out":578,"duration_ms":5651,"temperature":1.0,"reasoning_tokens":509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:30:13.210499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PIMAEX-Communication ($\\beta$) and PPO+RND on a second, independently designed partially observable coordination task with sparse rewards, using the same hyperparameter search budget and seeds; if $\\beta$ does not exceed RND in mean team return or variance reduction, the central empirical claim fails to generalize beyond its single environment.","supporting_citations":[{"cited_title":"A., Strouse, D., Leibo, J","cited_arxiv_id":null,"evidence_quote":"Supplies the $\\alpha$ policy-influence term, the discrete communication channel, and the counterfactual marginalization used in PIMAEX-Communication."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the $\\gamma$ value-influence term (Value of Interaction) that the generalized reward subsumes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides random network distillation, the intrinsic curiosity baseline and the RND mechanism used for intrinsic rewards in PIMAEX."},{"cited_title":"Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T","cited_arxiv_id":null,"evidence_quote":"The sequential social dilemma framework that the Consume/Explore environment instantiates."}],"review_version":1}