{"id":"e74df786-5579-4257-b439-d40280cb6026","arxiv_id":"2505.08630","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ISA uses mutual information between actions and state changes to define each agent's influence scope, then uses it for credit assignment and count-based exploration in sparse-reward MARL.","lead":"In this paper, the authors propose ISA, an algorithm for sparse-reward multi-agent reinforcement learning that identifies which parts of the environment each agent can influence and uses that knowledge to assign credit and guide exploration. The method is tested on StarCraft and particle tasks, where it outperforms several baseline MARL algorithms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Influence scopes are estimated once from random-policy transitions but used for the entire training run; if the joint distribution shifts, the scopes may be wrong for the policies that actually receive credit.","rationale":"The reader identifies the MI estimate and fixed scopes as the weakest assumption. I agree and sharpen it: the issue is not only finite-sample estimation error, but distribution shift. Eq. 4 is computed from random-policy transitions, while the scopes gate credit assignment and exploration for entirely different learned policies. Because Algorithm 1 computes scopes once (Line 4) and never refreshes them, any state-dependent action influence that emerges only under goal-directed behavior is missed by construction. This directly threatens the central claim of improved sample efficiency, since Eq. 7 and Eq. 10 hard-gate rewards and bonuses on scope membership. The paper does provide open-sourced code and ablations showing the method works better than no-scope variants, which is genuine evidence, but it does not establish that the fixed scopes are correct or stable under policy-induced distribution shift. A specific empirical check can settle this: recompute scopes during training and compare with the initial scopes, while also testing a periodically-updated variant. Until that is done, the conditional verdict is appropriate; I do not see grounds to reject the paper, because the method is coherent, code is available, and the worry may well fail empirically.","tokens_in":13206,"tokens_out":1871,"duration_ms":21955,"concrete_test":"Run 2s vs 1sc (or 3m) and recompute the influence scopes D_i and D(a_i) at regular intervals during training (e.g., every 25k environment steps) using the same estimator, binning, and delta=0.3 on newly collected transitions. Report the Jaccard similarity between the initial scopes and the recomputed scopes, and train one ISA variant that updates scopes periodically. If scopes change materially (Jaccard below about 0.8) or if periodic updating changes final performance by more than a small margin, the fixed-scope assumption in Algorithm 1 Line 4 is a load-bearing weakness; if scopes remain stable across training, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim depends on Definition 1 (Eq. 4): D(a_i) = {k | I(Delta s_k; a_i | a_{-i}) > delta}, with I estimated from transitions collected while agents act randomly or with early exploration policies. Algorithm 1 computes D(a_i), D_i, D_c, and D^(i-c) once in Line 4 and never updates them. But mutual information is a property of the joint distribution of (Delta s, a_i, a_{-i}), not an environment constant. As training progresses, policies change and the state distribution shifts; state-dependent action effects that are nearly invisible under random behavior can become decisive under goal-directed behavior, and vice versa. The paper provides no argument or experiment showing that scopes estimated from a few thousand random transitions remain valid for the policies that are later trained. This matters because Eq. 7 and Eq. 10 use the scopes as hard gates: if D(a_i) incorrectly excludes a dimension that the current action actually influences, the agent receives no credit for progress on that dimension; if D(a_i) incorrectly includes a dimension, the agent is rewarded for changes it did not cause. The ablation 'ISA w/o influence scope' shows the method fails when the scope is removed, but that does not demonstrate that the fixed scopes are the correct or stable ones; it only shows that some restriction of the goal/counting space helps in the tested tasks. The deferred Appendix C.3 reportedly contains significance and stability tests, but they are not included in the main text, so the core stability premise remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ISA (Influence Scope of Agents), a method for sparse-reward cooperative MARL that combines credit assignment and exploration. ISA defines the influence scope of an action as the set of state dimensions k for which the conditional mutual information I(Δs_k; a_i | a_{-i}) exceeds a threshold δ (Definition 1, Eq. 4), and the influence scope of an agent as the union of the scopes of its actions (Definition 2). These scopes are used to decompose a global goal into per-agent individual goals (Definition 3) and to split each individual goal into a common segment (jointly influenced by all agents) and a special segment (Definition 4). The credit-assignment rule in Eq. 7 gives an agent credit for progress on the common segment only if its action's scope intersects the common scope, while the exploration bonus in Eq. 10 gates counting-based novelty bonuses in the same way. Algorithm 1 computes the scopes once from random transitions (N = 2,000 or 10,000) and never updates them. Experiments on SMAC and MPE, with ablations and an interpretability example, are used to claim significant improvements over state-of-the-art baselines.","tokens_in":13538,"tokens_out":4957,"duration_ms":54270,"significance":"If the proposed approach is correct, it addresses a real and important problem in sparse-reward MARL: automatically decomposing global goals into interpretable, per-agent goals and assigning credit without dense reward shaping. The information-theoretic definition of influence scope is a conceptually clean way to connect actions to state dimensions, and the idea of restricting exploration to scopes is plausible and potentially useful. The paper also releases open-source code, which supports reproducibility. However, the central claims currently rest on empirical evidence whose key validity conditions are neither proved nor adequately tested: the stability of scopes computed from random transitions under later trained policies, the reliability of the MI estimates from limited samples, and the statistical significance of the reported improvements. The ablations do support the usefulness of the scope-based restriction and the two credit-assignment mechanisms, but they are reported on a single task. Overall, the paper presents a promising framework that needs additional evidence and analysis before the central claims can be accepted.","major_comments":[{"comment":"The influence scopes D(a_i) and D_i are computed once from transitions collected under random or early exploration policies and are never updated during training. Mutual information is a property of the joint distribution of (Δs_k, a_i, a_{-i}), not an environment constant; as policies change, the relevant joint distribution shifts. Since Eq. (7) and Eq. (10) use the scopes as hard gates, an incorrect scope can either withhold credit for genuinely influenced dimensions or grant credit for dimensions the agent does not influence. The paper provides no argument or experiment showing that scopes estimated from a few thousand random transitions remain valid for the goal-conditioned policies being trained. I recommend adding an empirical stability check (e.g., recomputing scopes at several points during training and measuring agreement) or an intervention-based validation on at least one benchmark, and discussing under which conditions the random-policy scopes are provably equal to the scopes under the training distribution.","section":"Section 4, Definition 1/Eq. (4); Algorithm 1, Line 4"},{"comment":"The threshold δ is fixed at 0.3 based on a reported 'workable range' [0.15, 0.45], but the other introduced hyperparameters (λ, α1 = β1, α2 = β2, N) are tuned per task, and in the SMAC domain α1 = β1 = 0, which disables the special-segment credit in Eqs. (7) and (10). The main text does not report sensitivity analysis; the reader is referred to Appendix C.3, which is not included in the manuscript under review. Because the central benefit of ISA depends on the quality of the influence scopes and on the balance between common and special segment rewards, the paper should present sensitivity results in the main text (or at least in an accessible appendix) and justify that the per-task tuning does not inadvertently favor ISA over baselines.","section":"Section 6, Hyperparameters"},{"comment":"The main text claims that ISA 'significantly outperforms' baselines based on learning curves with 5 seeds and without reporting any statistical test. Significance testing is deferred to Appendix C.3, which is not available for review. Since the paper's central assertion is empirical, the main text should include basic statistics for the key environments (e.g., mean ± standard deviation over seeds and a paired test at a fixed environment step), at least for the main comparison curves.","section":"Section 6, Results, Fig. 1"},{"comment":"The estimator of the conditional mutual information is described as randomly sampling combinations of a_{-i} and averaging the per-combination MI values. With N = 2,000 transitions and multi-agent action spaces, many a_{-i} combinations will have very few or no samples, so the estimated I(Δs_k; a_i | a_{-i}) may have high variance or bias. This directly affects the correctness of Definition 1. The paper should provide a bias/variance analysis or validate the estimator on a synthetic setting with known conditional independence, and should report the number of distinct a_{-i} combinations actually observed in the experiments.","section":"Section 5.1, Eq. (3)"}],"minor_comments":[{"comment":"'maximize the excepted accumulated reward' should read 'maximize the expected accumulated reward'.","section":"Section 3, Preliminaries"},{"comment":"The text says 'Fig .3(b) illustrates this scenario. At the time step of this screenshot...', but Fig. 3 shows heat maps, not a screenshot; the screenshot appears in Fig. 4(b). Please correct the figure reference.","section":"Section 6, Interpretability"},{"comment":"The final combination line 'ri + β2 r' is ambiguous because r_i is not explicitly defined as R_i+(s,a_i,s'); please define r_i+ consistently.","section":"Section 5.1, Eq. (10) and following text"},{"comment":"The phrase 'can can influence' appears in the goal-decomposition paragraph; please fix the duplicate.","section":"Section 4, Definition 3 and Remark 3"},{"comment":"References [Liu et al., 2021a] and [Liu et al., 2021b] have the same title and page numbers, which suggests a typo or unintended duplication. Please verify and correct.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript references Appendix C.3–C.5 for significance tests, hyperparameter sensitivity, and time consumption, but these appendices are not included in the reviewed version. If they are available in the full submission, the authors should be asked to include them in the review package and to summarize the key results in the main text. The per-task hyperparameter tuning (λ, α/β, N) is extensive; I recommend asking for a table of all settings and a description of the selection procedure, so that the reader can assess whether the comparison is fair."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Han, Dastani, and Wang have written a paper that deserves a real referee, not a desk rejection. The novelty is genuine: influence scope, defined as the set of state dimensions whose change is conditionally mutually informative about an agent's action, is a clean way to delimit individual goals in cooperative MARL. The credit-assignment rule in Eq. 7, which gates common-segment reward on whether the action touches the common scope, is a simple and interpretable idea. The counting-based exploration restricted to the scope (Eq. 10) is a natural extension. The ablations are well-designed: they show that removing the scope, the segmentation, or the gating rule each hurts, which supports the components. The code is open, and the comparison includes strong baselines (MASER, CMAE, FoX, HMASD). The heat maps in Fig. 3 are a nice sanity check for why conditioning on other agents' actions matters.\n\nThe main soft spot is exactly what the stress-test flags: scopes are computed once from a few thousand random-policy transitions (Line 4 of Algorithm 1) and then used for the entire training run. Mutual information is a property of the joint distribution, not just the environment. The paper's Remark 1 claims the scope is an 'inherent property of the environment,' but that needs evidence. The deferred Appendix C.3 reportedly contains stability and significance tests, but they are not in the main text of this version, and the core stability claim is too load-bearing to leave in an appendix. The ablation 'w/o influence scope' shows that restricting the goal/count space helps in these tasks, but it does not show that the fixed scopes are the correct ones under the trained policies. That is a fixable issue: the authors could re-estimate scopes periodically, or at least plot the scopes' overlap across training, or show that results are insensitive to the choice of delta within the stated range.\n\nThe hyperparameter story is honest but mixed. Delta is set to 0.3 with a claimed workable range, but lambda, alpha1/beta1, and alpha2/beta2 are tuned per task (lambda is 0, 10, or 50 across environments). That is more tuning than I would like, but not disqualifying for a methods paper. The reported five-run learning curves show large margins, but significance testing is deferred to an appendix that is not present in this submission.\n\nOverall: the central idea is well-motivated, the implementation is careful, and the empirical story is plausible. The stability of the influence scopes is a genuine question that the authors must address in the main text before the claim of 'significantly outperforms' is fully credible. I would send this to peer review with a request to move the stability analysis forward and to make the appendix available. It is a useful paper for anyone working on sparse-reward MARL, and I would cite it once the stability question is settled.","headline":"Genuinely novel credit-assignment idea with solid ablations and open code; the unverified stability of one-shot influence scopes is the main caveat, but the paper merits a serious referee.","tokens_in":14054,"tokens_out":3056,"would_cite":true,"duration_ms":29124,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-agent influence scopes, measured by mutual information, drive credit assignment and exploration in sparse-reward MARL.","keywords":["multi-agent reinforcement learning","sparse rewards","credit assignment","exploration","mutual information","influence scope","goal-conditioned reinforcement learning"],"falsifier":"Run ISA on a sparse-reward MPE task where one agent's action only affects a state dimension after another agent has reached a prerequisite landmark, and re-estimate the influence scopes midway through training. If the newly computed scopes differ from the initial scopes in a way that changes which dimensions are credited or explored, then the central assumption that scopes can be fixed before training is false; equivalently, a plot of $D_i$ over training that grows to include previously excluded dimensions would falsify the once-only computation.","tokens_in":13017,"feed_emoji":"🎯","tokens_out":7554,"duration_ms":64323,"temperature":0.7,"pith_summary":"This paper tries to establish that in cooperative multi-agent reinforcement learning with sparse rewards, agents should be given individual goals defined by their 'influence scope' — the set of state dimensions whose changes depend on their action, measured by conditional mutual information. The authors argue that this scope solves two problems simultaneously: credit assignment, because an agent is rewarded only for progress on state segments its action can influence, with jointly influenced segments handled through a common/special split; and exploration, because count-based bonuses count only the state projections inside the scope instead of the full high-dimensional state. The algorithm, ISA, computes the scopes once from a few thousand random transitions, then trains exploration policies and goal-conditioned policies. If the claim is right, sparse-reward tasks can be solved without dense reward shaping and with an interpretable per-agent decomposition of global goals.","feed_headline":"Influence scopes assign credit and guide exploration in sparse MARL","feed_subtitle":"New ISA method rewards agents only for state changes they can cause and counts only scoped states for exploration.","key_machinery":"The central object is the influence scope, an index set of state dimensions defined by $D(a_i)=\\{k \\mid I(\\Delta s_k; a_i \\mid a_{-i})>\\delta\\}$, computed from transitions collected under random policies. The dependence measure is estimated by equal-width binning of the state change $\\Delta s_k$, binarization of the action $a_i$, and averaging the conditional mutual information over sampled values of the other agents' actions. This scope is the load-bearing device: it defines each agent's individual goal as the projection of a global goal onto $D_i$ (Definition 3), splits that goal into a common segment $D_c=\\cap_i D_i$ and a special segment $D_i\\setminus D_c$ (Definition 4), and then gates both the goal-conditioned reward in Eq. (7) and the exploration bonus in Eq. (10) on whether the current action can influence the common segment ($D(a_i)\\cap D_c\\neq\\varnothing$).","core_discovery":"The central discovery is that a thresholded conditional mutual information, $D(a_i)= \\{k \\mid I(\\Delta s_k; a_i \\mid a_{-i}) > \\delta\\}$, defines an agent's influence scope and can drive both credit assignment and exploration in sparse-reward MARL. The paper shows how to estimate this quantity by discretizing state changes, binarizing actions, and averaging over other agents' actions, and then uses the agent-level scope $D_i=\\cup_{a_i}D(a_i)$ to decompose a discovered global goal into per-agent sub-goals (Definition 3). The reward rule of Eq. (7) gives an agent credit from the common segment only when its current action can influence it, and otherwise only from its special segment, which prevents other agents' actions from corrupting its feedback. The exploration rule of Eq. (10) applies the same gating to count-based novelty bonuses restricted to the scoped projections (Eqs. (8)-(9)). In experiments on super-sparse SMAC and MPE tasks, the paper reports that ISA outperforms IPPO, QMIX, COMA, MASER, CMAE, FoX and HMASD in sample efficiency and final performance, and ablations show that removing the scope-based credit assignment or exploration degrades learning.","pith_inferences":["Because Algorithm 1 computes influence scopes once before training, a natural extension is periodic re-estimation; if scopes drift as policies change, the fixed-scope version could miss newly relevant state dimensions.","The mutual information criterion captures immediate influence only; the paper's manual 2-step adjustment for 2s vs 1sc hints that a general multi-step influence measure would extend ISA to tasks with long action delays.","The threshold $\\delta$ is hand-tuned; an adaptive threshold or a significance test on the mutual information estimate would remove a domain-dependent hyperparameter and make the method easier to apply.","The hard gating in Eq. (7) could be replaced by a soft weighting proportional to $I(\\Delta s_k; a_i\\mid a_{-i})$, which might smooth the reward signal when influence estimates are noisy near the threshold."],"forward_implications":["Sparse-reward cooperative tasks become learnable without handcrafted dense reward shaping, because intrinsic rewards are derived from state dimensions each agent can actually influence.","The common/special segment split prevents unstable feedback: an agent is rewarded for the jointly influenced segment only when its current action affects it, so other agents' actions cannot corrupt its credit signal.","Exploration is more efficient because count-based bonuses count only scoped projections of the state, avoiding the curse of dimensionality in high-dimensional multi-agent state spaces.","Credit assignment becomes interpretable: the if-else rule in Eq. (7) yields a clear explanation of which actions are credited for which state changes, as demonstrated in the 8m example where agent 7 receives less reward while not contributing to the common segment.","The overhead of computing scopes is small because they are estimated from 2,000 to 10,000 transitions, and the method is applicable across SMAC and MPE benchmarks with a single threshold $\\delta=0.3$."],"supporting_citations":[{"why":"Supplies the equal-width binning used to estimate mutual information from continuous state changes.","marker":"[Kraskov et al., 2004]"},{"why":"Provides the goal-conditioned intrinsic reward paradigm that ISA builds on and the caution that treating wrong state information as a goal is harmful.","marker":"[Colas et al., 2022]"},{"why":"Count-based exploration that Eqs. (8) and (9) adapt by restricting counting to influence-scope projections.","marker":"[Tang et al., 2017]"},{"why":"Justifies training toward multiple discovered goals via the universal value function approximator / multi-goal RL paradigm.","marker":"[Schaul et al., 2015]"},{"why":"HMASD, the state-of-the-art sparse-reward MARL baseline that the paper must outperform in SMAC.","marker":"[Yang et al., 2024]"},{"why":"MASER, a subgoal-based sparse-reward baseline with distance-based intrinsic rewards similar to ISA's.","marker":"[Jeon et al., 2022]"},{"why":"IPPO, the policy-loss backbone used for both the exploration and goal-conditioned policies.","marker":"[De Witt et al., 2020]"}],"fun_headline_variants":["Scope rewards by influence: credit and exploration without dense feedback","Give credit where influence lies: sparse MARL via scope","Only reward what you influence: scoped credit and exploration for agents","Influence scope: sparse rewards, precise credit, focused exploration","Agent influence scopes: credit and exploration in sparse reward MARL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the influence measure computed once from a few thousand random transitions, cut off at a hand-tuned threshold, correctly names the state components each agent's action can ever affect, and that these names stay correct throughout training even though they are never updated.","fun_headline_variants_meta":{"raw":{"variants":["Scope rewards by influence: credit and exploration without dense feedback","Give credit where influence lies: sparse MARL via scope","Only reward what you influence: scoped credit and exploration for agents","Influence scope: sparse rewards, precise credit, focused exploration","Agent influence scopes: credit and exploration in sparse reward MARL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001334,"raw_usage":{"total_tokens":5429,"prompt_tokens":952,"completion_tokens":4477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":4391}},"tokens_in":568,"tokens_out":4477,"duration_ms":29199,"temperature":1.0,"reasoning_tokens":4391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:48:55.989658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ISA on a sparse-reward MPE task where one agent's action only affects a state dimension after another agent has reached a prerequisite landmark, and re-estimate the influence scopes midway through training. If the newly computed scopes differ from the initial scopes in a way that changes which dimensions are credited or explored, then the central assumption that scopes can be fixed before training is false; equivalently, a plot of $D_i$ over training that grows to include previously excluded dimensions would falsify the once-only computation.","supporting_citations":[{"cited_title":"Estimating mu- tual information","cited_arxiv_id":null,"evidence_quote":"Supplies the equal-width binning used to estimate mutual information from continuous state changes."},{"cited_title":"Autotelic agents with intrinsically motivated goal-conditioned reinforce- ment learning: A short survey","cited_arxiv_id":null,"evidence_quote":"Provides the goal-conditioned intrinsic reward paradigm that ISA builds on and the caution that treating wrong state information as a goal is harmful."},{"cited_title":"#exploration: A study of count-based exploration for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Count-based exploration that Eqs. (8) and (9) adapt by restricting counting to influence-scope projections."},{"cited_title":"Universal value function ap- proximators","cited_arxiv_id":null,"evidence_quote":"Justifies training toward multiple discovered goals via the universal value function approximator / multi-goal RL paradigm."},{"cited_title":"Hierarchical multi- agent skill discovery","cited_arxiv_id":null,"evidence_quote":"HMASD, the state-of-the-art sparse-reward MARL baseline that the paper must outperform in SMAC."},{"cited_title":"MASER: multi-agent rein- forcement learning with subgoals generated from experi- ence replay buffer","cited_arxiv_id":null,"evidence_quote":"MASER, a subgoal-based sparse-reward baseline with distance-based intrinsic rewards similar to ISA's."}],"review_version":1}