{"id":"efbbccf2-d70f-4d3d-99d7-d5b4de69f6b9","arxiv_id":"2501.11818","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"HGARL lets heterogeneous RL agents share action choices and model parameters, claiming large speed-ups on Atari, though the reported statistics use the best of three rules per game.","lead":"This paper extends group-agent reinforcement learning to heterogeneous agents, where A2C, PPO, and ACER agents share action suggestions and model parameters while learning on 43 Atari games. The headline speed-up claims are weakened by a best-of-three rule selection and by treating unreached single-agent targets as infinite speed-ups.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline percentages rely on per-game best-of-three rule selection and count censored 'inf' entries as infinite speed-ups; a fixed-rule reanalysis is needed.","rationale":"The reader's stated weakest_assumption, the identical action/state spaces and architecture requirement, is a real restriction but it is explicitly stated in Section 3 and limits the claimed generality rather than invalidating the central comparison under that restriction. The more load-bearing problem is in Section 4.2: the headline statistics are obtained by taking, for each game and agent type, the best performance among the three selection rules. Since no a priori rule-selection mechanism is given, the reported 96%/72%/41% figures cannot be attributed to any fixed variant of the proposed algorithm. The 'inf' entries are treated as arbitrarily large speed-ups, although they only mean that the single agent did not reach the group agent's score within its finite budget; a longer single-agent run would convert some of them into finite, possibly much smaller, speed-ups. The fine-tuning of the Combo threshold phi on the test games adds further selection bias. A re-tabulation with a fixed rule and censored inf would settle whether the quantitative claim is real. Because this concern directly attacks the central empirical claim and the paper provides no alternative evidence for it, the reader's REJECT verdict remains appropriate.","tokens_in":22796,"tokens_out":6189,"duration_ms":69679,"concrete_test":"Re-tabulate Tables 1 and 2 from the existing Table 6 data using a fixed rule chosen before seeing test results: use Combo only (the only rule with model adoption), or select the rule on a training subset of games, not on each test game. Also re-estimate speed-up with 'inf' treated as right-censored by replacing inf with T_max/TG, where T_max is the total single-agent training budget, and report the fraction with lower-bound speed-up >100. If the fixed-rule and censored fractions fall materially below 96%/72% (e.g., the Combo-only ACER column in Table 6 contains many entries below 1), the abstract's aggregate claim depends on oracle rule selection and cannot be accepted as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline figures (96% speed-up, 72% >100x, 41% reward within 5% time) are compiled from Table 2, whose note states: 'for each agent type and test case (game), we report the performance of the rule that gives the best performance among the three rules.' Each game therefore contributes the maximum over PA, PM, and Combo, and the percentages do not describe any single concrete algorithm a user could deploy. No rule-selection mechanism is provided for a new game, so the claimed method is effectively an oracle over three rules. This is compounded by the treatment of 'inf' in Table 6: an infinite speed-up means the single agent never reached the group agent's score within the finite training budget, yet those entries are counted as successes and as >100x speed-ups (18/43 A2C, 29/43 ACER, 6/43 PPO entries in Table 2). These are censored observations, not measured infinite improvements. Appendix A also reports that the Combo threshold phi was 'fine-tuned' on the test games, so even the Combo-only comparison has test-set selection. The central quantitative claim is therefore not established by the reported experiments as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HGARL, a framework for asynchronous knowledge sharing among reinforcement learning agents that run different algorithms (A2C, PPO, and ACER) in separate environments. Knowledge is shared as action suggestions from peer policies and as full policy/value model parameters. Three action-selection rules are defined (probability addition, probability multiplication, and reward-value-likelihood combination, called Combo), along with a model-adoption rule. Experiments on 43 Atari games are used to claim that 96% of the 129 agents achieve a learning speed-up, 72% learn over 100 times faster, and 41% reach a higher reward within 5% of the time steps needed by a single agent. The central quantitative claims are computed by selecting the best of the three rules per agent and game, by counting 'inf' speed-ups as successes, and by fine-tuning the Combo threshold on the test games.","tokens_in":22985,"tokens_out":5321,"duration_ms":51888,"significance":"If validated, HGARL would be a practically relevant contribution: it would show that agents running different RL algorithms can improve one another's sample efficiency by sharing policies and models asynchronously without a shared environment. The experimental scale is substantial (43 games, three agents, four seeds, over 10M core hours), and the idea of combining heterogeneous actor-critic algorithms through action-selection and model-adoption mechanisms is plausible. However, the paper does not provide code, proofs, or a parameter-free derivation; the contribution is empirical. As reported, the evaluation does not establish the headline claims because of the issues detailed in the major comments. The underlying direction is worth pursuing, but the current evidence is insufficient for the strength of the claims.","major_comments":[{"comment":"The notes to Table 1 and Table 2 state that for each agent type and game the authors report the performance of the rule that gives the best performance among PA, PM, and Combo. Consequently, the headline numbers (96% speed-up, 72% >100x, 41% reward within 5% time) are maxima over three rules, and no single rule or concrete algorithm is evaluated. Since the paper does not supply a rule-selection mechanism for a new game, these percentages describe an oracle rather than a deployable method. Please report per-rule statistics separately, and if a combined method is claimed, define how the rule is chosen without access to the test game.","section":"§4.2, Tables 1 and 2"},{"comment":"The 'inf' entries in Table 6 mean that the single agent never reached the group agent's highest performance within the finite training budget. In Table 2 these are counted in the 'inf' bin and treated as speed-ups greater than 100x. This is a censored observation, not an infinite speed-up. For instance, the ACER column in Table 2 contains 29 of 43 'inf' entries, which heavily inflates the aggregate statistics. The authors should analyze performance at a fixed time budget (for example, comparing final scores at equal numbers of steps) and report cases where the single agent never reached the group agent's score as 'not reached', not as >1000x or infinite.","section":"§4.2, Table 2 and Table 6"},{"comment":"The threshold phi in Eq. (8) is described as 'fine-tuned' on the test games in Appendix A. This is test-set selection, so the Combo-rule results cannot be interpreted as a validation of the method's generalization. The authors should either select phi on a validation split before evaluating on the test games, or provide a sensitivity analysis over phi values chosen a priori and show that the reported improvements are not artifacts of a particular tuned threshold.","section":"§4.1 and Appendix A"},{"comment":"The treatment of 'nan' values in Table 6 is not explained in the statistics. The paper notes that 'nan values mean the group agents never reached a satisfactory level of performance', but it does not specify whether these are counted as failures, excluded from the percentages, or assigned to one of the speed-up bins. Since the totals in Table 2 sum to 43 for each agent, at least some 'nan' entries must have been omitted or absorbed into an existing bin; this handling should be stated explicitly and justified.","section":"§4.2, Table 1, Table 2, and Table 6"}],"minor_comments":[{"comment":"The symbol phi is introduced in Eq. (8) without stating that it is a threshold on the negative log likelihood; the definition appears only in the following paragraph. Please define the threshold at its first use.","section":"§3.2, Eq. (8)"},{"comment":"In Figure 1, the label 'aStep' appears to be a fragment; it should probably be 'Action Step' or 'Apply'. Also, Algorithm 1 uses 'AN D' with a space in lines 14-16, which should be 'AND'.","section":"Figure 1 and Algorithm 1"},{"comment":"The assumption of identical state and action spaces and identical policy/value architectures is stated in Section 3, but the title and abstract do not qualify the meaning of 'heterogeneous agents' beyond algorithmic differences. Please scope the terminology throughout the paper to avoid overclaiming generality.","section":"§3, introductory paragraph"},{"comment":"The paper states that the experiments consumed over 10M core hours but does not provide a precise breakdown per run. A short table listing the number of runs, environment steps per run, and hardware details would improve transparency and reproducibility.","section":"§4.1, experiment setting"}],"recommendation":"major_revision","confidential_remarks":"The current manuscript's central quantitative claims are not supported by the evaluation as reported. The best-of-three rule selection, the counting of censored 'inf' observations as successes, and the test-set tuning of phi are all fixable in principle, but they require a substantial reanalysis. I would encourage the authors to resubmit after performing a fixed-rule analysis with a validation split and a proper treatment of censored measurements; with those changes, the paper could become publishable. The reviewer is not questioning the inherent plausibility of the proposed framework, only the validity of the evidence as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kaiyue,\n\nThe paper is a genuine extension of GARL: it moves from homogeneous gradient-sharing to heterogeneous agents (different algorithms) by sharing action choices and model parameters, with three action-selection rules (PA, PM, Combo) and a model-adoption mechanism. The setting is new relative to the existing GARL work, and the experimental effort is serious: 43 Atari games, three agents, four seeds, and detailed learning curves. The authors also explain the behaviour of the rules, e.g., why Combo is conservative on Alien.\n\nThe problem is the evaluation. The headline claims (96% speed-up, 72% >100x, 41% within 5% time) are compiled from Table 2, which takes, for each agent and game, the best of the three rules. The paper is transparent about this in the table note, but it means the numbers do not describe any single method a user could run. Without a rule-selection mechanism for a new game, the method is effectively an oracle over three rules, and the abstract presents it as one method. That is a mismatch between claim and evidence.\n\nAlso, the 'inf' speed-ups are censored observations: the single agent never reached the group agent's score within the budget, yet they are counted as successes and as >100x. That inflates the percentages. And Appendix A shows the Combo threshold phi was fine-tuned on the test games, so even the Combo-only comparison has test-set selection. The authors state this openly, but it still weakens the central quantitative claim.\n\nThere are also no baselines against existing knowledge-transfer or population-based methods, so it's hard to say whether the gains come from the specific rules or just from sharing anything at all.\n\nThe architecture-sharing assumption (same action/state spaces, same policy architecture) limits 'heterogeneous' to differences in learning algorithm, which is narrower than the title suggests but still a reasonable scope. The authors state it clearly.\n\nNone of this kills the idea. The setting is real, the rules are sensible heuristics, and the experiments are extensive. But the quantitative claims need to be re-run with a fixed rule per agent or a proper selection rule, censored observations handled as such, and hyperparameter selection held out. With that, the paper could be a useful contribution.\n\nMy take: send it to review, but expect heavy revision on the evaluation. The novelty and effort justify referee time; the current numbers do not.","headline":"Real extension of GARL to heterogeneous agents, but the headline speed-up numbers describe an oracle over three rules, not a single deployable method.","tokens_in":23558,"tokens_out":3008,"would_cite":false,"duration_ms":27277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Heterogeneous RL agents that share action choices and adopt each other's models learn dramatically faster than solo agents, with 96% of 129 examined Atari learners seeing a speed-up.","keywords":["heterogeneous agents","group-agent reinforcement learning","action selection rules","model adoption","knowledge sharing","Atari 2600","sample efficiency","actor-critic algorithms"],"falsifier":"Recompute the speed-up statistics from the paper's Table 6 using one fixed selection rule for all agents (say Combo only) instead of the best of the three rules per game; if the 96% and 72% headline percentages fall sharply, the central quantitative claim is an artifact of per-game rule selection.","tokens_in":22555,"feed_emoji":"🎮","tokens_out":6597,"duration_ms":58855,"temperature":0.7,"pith_summary":"This paper advances group-agent reinforcement learning to a heterogeneous setting: each agent in a group runs a different actor-critic algorithm (A2C, PPO, or ACER), acts in its own copy of the environment, and asynchronously exchanges two kinds of knowledge—policy/value model parameters and the accumulated reward score per episode. The claim is that using this shared knowledge, each individual agent learns faster and often reaches a higher final reward than if it trained alone. On 43 Atari 2600 games this yields, across 129 agents, a learning speed-up for 96% of them, with 72% learning 100 times faster and about 41% reaching a higher reward in less than 5% of the time steps a single agent needs. The reason this would matter is that it offers a simple, algorithm-agnostic way to accelerate RL training in settings where several agents independently solve similar tasks.","feed_headline":"Sharing moves and models speeds up 96% of RL agents","feed_subtitle":"On 43 Atari games, A2C, PPO and ACER learners often reach top scores 100x faster in a group.","key_machinery":"The load-bearing mechanism is the combination of three action-selection rules and a model-adoption step operating under a shared-interface assumption. The rules—probability addition (sum of the group's action probabilities), probability multiplication (product of those probabilities), and the Combo rule (which restricts attention to agents with higher accumulated reward, picks the one with the highest accumulated next-state value, and applies a negative-log-likelihood confidence threshold)—turn a set of peer-suggested actions into one action per step. Model adoption, applied only with the Combo rule, replaces an agent's own policy and value networks with a peer's when that peer's policy dominated the agent's last batch of choices and holds the highest accumulated reward. The enabling assumption that backs the entire scheme is that all agents in a group share the same state space, action space, and network architecture, so parameters can be interchanged freely.","core_discovery":"The central discovery is a set of group-learning mechanisms that let heterogeneous RL agents improve one another without synchronizing. At each time step an agent collects the action suggestions of all group members, then selects one action using either probability addition, probability multiplication, or the Combo rule (which filters suggestions by accumulated reward, next-state value, and action confidence). When Combo is used, an agent may also adopt the entire policy and value model of a peer whose suggestions it has been following for at least half of a training batch and who has the highest accumulated reward. The paper reports that on 43 Atari games the resulting HGARL agents reach their best scores in a fraction of the steps a solo agent needs: 96% of the 129 examined agents achieve a speed-up, 72% learn at least 100 times faster, and about 41% achieve a higher reward score within 5% of the time steps required by single-agent learning.","pith_inferences":["Beyond the paper: because the headline figures choose the best rule per game, a fixed-rule deployment would likely show lower average speed-up; a natural extension is an online rule-selection meta-policy that picks between PA, PM, and Combo from observed group performance.","Beyond the paper: the shared-architecture assumption means 'heterogeneous' is limited to algorithm differences; a variant that shares only action advice, without model parameters, could test whether the speed-up comes mainly from better action choices or from parameter adoption.","Beyond the paper: the Combo rule's threshold is tuned once across games; one could expect per-agent or per-game threshold adaptation to improve the conservative-filtering failures seen on games like Alien."],"forward_implications":["A2C, PPO, and ACER agents trained as a group can match or beat their solo performance using a small slice of the environment steps, with 41% of tests reaching a higher final reward in at most 5% of the steps.","The speed-up is not limited to one rule: the best rule varies by game, and the reported 96% figure is the best-of-three per game, so a practical deployment needs a way to pick or switch rules.","Model adoption accounts for the extreme speed-ups, including the 'inf' entries where solo agents never reach the group performance; without adoption, only action selection remains.","The group protocol is asynchronous and each agent keeps its own environment, so it is compatible with decentralized training workloads."],"supporting_citations":[{"why":"Supplies the A2C, PPO, and ACER implementations used for all agents and the single-agent baselines.","marker":"Dhariwal et al. [2017]"},{"why":"Provides the Atari 2600 Arcade Learning Environment used as the 43-game benchmark.","marker":"[Bellemare et al., 2013]"},{"why":"Introduces the original GARL concept that this work extends from homogeneous gradient sharing to heterogeneous model sharing.","marker":"[Wu and Zeng, 2023]"},{"why":"Defines the A2C algorithm used as one of the three heterogeneous agent types.","marker":"[Mnih et al., 2016]"},{"why":"Defines PPO, the second agent type.","marker":"[Schulman et al., 2017]"},{"why":"Defines ACER, the third agent type.","marker":"[Wang et al., 2017]"}],"fun_headline_variants":["Group learning makes 96% of RL agents faster","Heterogeneous RL agents share advice to speed up learning","Cooperative RL agents learn 100x faster on Atari","Agent groups adopt better policies to accelerate learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole sharing scheme rests on every agent in the group using the same state space, action space, and policy-network architecture; if a real group mixes architectures or action spaces, the action-selection and model-adoption steps can no longer be executed.","fun_headline_variants_meta":{"raw":{"variants":["Group learning makes 96% of RL agents faster","Heterogeneous RL agents share advice to speed up learning","Cooperative RL agents learn 100x faster on Atari","Agent groups adopt better policies to accelerate learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1455,"prompt_tokens":925,"completion_tokens":530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":541,"tokens_out":530,"duration_ms":5022,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:49:49.388224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the speed-up statistics from the paper's Table 6 using one fixed selection rule for all agents (say Combo only) instead of the best of the three rules per game; if the 96% and 72% headline percentages fall sharply, the central quantitative claim is an artifact of per-game rule selection.","supporting_citations":[{"cited_title":"Group-agent reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Introduces the original GARL concept that this work extends from homogeneous gradient sharing to heterogeneous model sharing."}],"review_version":1}