{"id":"c816f8fd-9cfb-4ef6-8d16-2277fd8f58c0","arxiv_id":"2501.10938","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-expert action-advice method plus a blockchain model marketplace speeds up multi-agent reinforcement learning under sparse rewards and tolerates faulty experts.","lead":"This paper combines a multi-agent reinforcement learning trick, where trained expert models suggest actions during training, with a blockchain marketplace for sharing those models. The authors report faster learning and better tolerance of faulty experts in three simulated multi-agent tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Q-threshold in Algorithm 2 does not restore on-policy validity: expert-selected actions are stored and used in PPO updates without an off-policy correction, so the reported speedup may be an artifact of biased gradient estimates.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: Algorithm 2 violates the on-policy assumption that makes the PPO clipped surrogate in Eq. (1) an unbiased estimator. This is the most important issue because the paper's central claim is that expert suggestions produce faster learning under sparse rewards while remaining on-policy; if the Q-threshold does not preserve on-policy validity, the reported learning-speed gains could be an artifact of biased gradient updates rather than evidence of better exploration. The concern is technical and specific: the action selected by the expert and stored at line 14 is not sampled from the agent's policy, so the stored data are off-policy, and no importance-weight correction is applied. The Q filter only ensures the selected action has nontrivial probability under the current policy, which is not the same as sampling from that policy. The paper provides no theoretical guarantee, no off-policy-corrected baseline, and no ablation that isolates the contribution of the Q mechanism. The proposed concrete test directly isolates this mechanism by keeping expert input but restoring on-policy sampling; if the gain persists, the concern is retired, and if it disappears, the central claim needs re-framing. Other weaknesses (no seeds, no error bars, possibly unfair FRL comparison) are secondary and were already captured in the reader's CONDITIONAL verdict, so the verdict should remain UNCHANGED rather than being moved to a different category.","tokens_in":19155,"tokens_out":4348,"duration_ms":51564,"concrete_test":"Re-run the A3W2 experiment of Fig. 6 with a single change to Algorithm 2: at line 14, instead of a = argmax(ExpActions), sample a_j^k from the truncated categorical P_k restricted to actions with P_k[x] > Q, with the expert's ranking used only to add a fixed logit bonus before sampling (so all stored actions are on-policy samples from a modified policy). If the episodic-length curves over 2×10^7 steps remain close to the reported MEDC curve, the on-policy bias is not the driver and the concern is retired. If the curve drops toward the MDRL-Sparse baseline, the reported speedup is an artifact of the off-policy update, and the Q-threshold justification in Section III-B is empirically false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B and Algorithm 2 (lines 7-14) let the chosen expert select the action actually stored and used in the PPO update: after thresholding actions with P_k[x] < Q, the agent takes a = argmax(ExpActions), not a sample from P_k. PPO's clipped surrogate in Eq. (1) is an unbiased estimator only when the stored actions are drawn from π_θ_old; here the behavior policy is a mixture of π_θ_old (with probability 1-RE) and an expert-dependent, deterministic selection (with probability RE). The Q condition merely restricts the accepted action to have P_k[x] > 0.05 under the current policy; it does not make the action distribution equal to P_k, so the probability ratio r_t(θ) in Eq. (2) is not a valid likelihood ratio and the gradient is biased. With Q=0.05 and a small discrete action space, early in training P_k is near-uniform and most or all expert actions pass the threshold, so the bias can be large at exactly the stage where Fig. 6 reports the largest speedup. The paper offers no off-policy correction (no importance sampling, no Retrace or λ-reweighting) and no ablation showing that the gain survives when the same expert suggestions are fed back through an on-policy sampling mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MEDC, a multi-agent RL training protocol in which an agent's PPO actor occasionally (with probability R_E per episode) follows actions suggested by externally shared expert or semi-expert models, subject to a probability threshold Q under the current policy. The same framework includes a consortium-blockchain/IPFS model-sharing system with smart contracts for reputation-based allocation. The authors evaluate MEDC on target localization, fleet coordination, and maze cleaning, comparing it with MDRL using sparse rewards, FRL, reward shaping, and IL-assisted RL, and report faster learning and resilience to faulty/malicious experts.","tokens_in":19428,"tokens_out":7502,"duration_ms":87502,"significance":"The practical direction is useful: rather than averaging models, agents can use heterogeneous expert models as action proposers, which sidesteps architecture restrictions and reduces the impact of poisoned models. Strong points include the explicit Algorithms 1 and 2, the concrete smart-contract data structures and gas-cost analysis, and evaluation across three different MDRL applications. However, the core algorithmic claim that the Q-threshold preserves PPO's on-policy update is not justified, and the empirical evidence lacks multiple seeds and error bars, so the quantitative outperformance claims are not yet established.","major_comments":[{"comment":"The threshold Q does not restore on-policy validity. In Algorithm 2, during expert episodes the action stored and used in the PPO update is a_t = argmax(ExpActions) after zeroing actions with P_k[x] < Q; it is not sampled from the agent's old policy π_θ_old. The clipped surrogate in Eq. (1) is an unbiased estimator of the policy-gradient objective only when the stored actions are drawn from π_θ_old, but here the behavior policy is a mixture of π_θ_old and an expert-dependent deterministic selection with probability R_E. Consequently r_t(θ) in Eq. (2) is not a valid likelihood ratio and the gradient estimate is biased. The condition P_k[a_t] > Q only bounds the probability of the selected action under the current policy; it does not make the action distribution equal to P_k. With Q = 0.05 and a small discrete action space, early in training P_k is near-uniform, so most expert suggestions pass the threshold and the bias can be large precisely in the phase where Fig. 6 reports the largest speedup. The paper offers no off-policy correction (importance sampling, Retrace, or similar) and no ablation in which the same expert suggestions are fed back through on-policy sampling. This issue must be resolved before the learning-speed claim is accepted.","section":"Section III-B, Algorithm 2, Eqs. (1)-(2)"},{"comment":"The empirical evidence for the central claim is based on single learning curves. Figures 6-11 show one trace per condition, and the text does not state the number of independent runs, standard deviations, confidence intervals, or statistical tests. Given that the differences between MEDC and the reward-shaping baseline in Fig. 8 are modest and the curves are visibly noisy, the reported 'outperformance' and 'resiliency' conclusions are not statistically supported. The experiments should be repeated over multiple random seeds and reported with means and error bars, and the resilience claim should include a measure of variance over the faulty-expert configurations.","section":"Section V-B, Section V-C, Figs. 6-11"},{"comment":"The method's two central hyperparameters, expert rate R_E = 0.1 and action threshold Q = 0.05, are fixed to single values with no sensitivity analysis. Section III-B describes qualitatively how Q should balance expert involvement and on-policy learning, but no experiment varies Q or R_E, and the resilience claim that MEDC is 'at worst' as good as MDRL-sparse is conditional on these values. Since both hyperparameters control how much expert behavior enters the policy update, the robustness of the reported gains is undemonstrated; a sweep over at least Q and R_E is needed.","section":"Table III, Section III-B (R_E and Q)"},{"comment":"The FRL comparison is set up in a way that may disadvantage the baseline by construction. In the target-localization benchmark, the global FRL model is formed by averaging models from eight users with deliberately different wall and agent counts (A1W0 through A3W3). This is a known failure mode for FedAvg-style aggregation, so the conclusion that MEDC 'outperforms FRL' is only established for a heterogeneous setting with naive aggregation, not as a general statement. The paper should either compare with an FRL variant designed for heterogeneous environments or explicitly scope the claim to naive FedAvg-style FRL.","section":"Section V-C"}],"minor_comments":[{"comment":"Algorithm 1 and Algorithm 2 use `i % H == 0` inside the inner episode loop, but `i` is the outer loop counter over episodes, not a count of environment timesteps; this makes the update schedule inconsistent with the text that H = 4000 timesteps per update. The pseudocode should be corrected.","section":"Algorithms 1-2"},{"comment":"Algorithm 2 does not define the fallback when no expert action satisfies P_k[x] >= Q; in that case all entries of ExpActions are zero and argmax is undefined.","section":"Algorithm 2"},{"comment":"The notation r_t is used both for the PPO probability ratio in Eq. (2) and for the reward in Eq. (3), which is confusing and should be disambiguated.","section":"Eqs. (2)-(3)"},{"comment":"Section V-D repeatedly says 'costumers' instead of 'customers'; this typo should be fixed throughout the two application descriptions.","section":"Section V-D"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper sits at the boundary between RL methodology and blockchain systems, and referees in both areas may want different emphases. The most serious issue is formally fixable: either weaken the on-policy claim and add an off-policy-corrected variant, or re-run with the same expert suggestions sampled on-policy. I would not reject the paper outright, because the system-level contribution (smart contracts, IPFS, reputation management) is concrete and the experimental setup, while needing seeds and error bars, is honest enough. I also note the heavy reliance on the authors' previous environment and method papers [7,28]; Section V should include the single-expert demonstration-cloning baseline from [28] to clarify the incremental contribution over prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is simple: instead of averaging model weights like FRL, let a pool of expert policies suggest actions during PPO training, filter those suggestions by a probability threshold Q under the agent's current policy, and then follow the most probable accepted action. That is a reasonable heuristic, and the multi-expert roulette selection plus the blockchain/IPFS allocation layer are genuinely new additions. The evaluation spans three environments, and the learning curves consistently favor MEDC over FRL, reward shaping, and IL-assisted RL. The resiliency experiment with random, biased, and malicious experts is a good practical check, and the result that performance at worst degrades to sparse-reward MDRL is believable given the low expert rate (RE=0.1).\n\nThe biggest soft spot is the on-policy claim. Algorithm 2 stores expert-chosen actions and feeds them to the PPO surrogate, so the probability ratio in Eq. (2) is not a valid likelihood ratio for those samples. The Q threshold just filters out actions with P_k[x] < 0.05; it does not make the behavior policy equal to π_θ_old. This is a real theoretical gap, and the paper explicitly claims the opposite. In practice PPO is often tolerant of some off-policy data, so the speedup may still survive, but the paper needs to either drop the on-policy framing and treat MEDC as a heuristic that injects off-policy bias, or add an off-policy correction. An ablation that feeds the same expert suggestions through an on-policy sampling mechanism would clarify whether the bias is driving the gains.\n\nThe experimental evidence is also thinner than the text suggests. No error bars, no multiple seeds, no code release, and the FRL baseline averages models across deliberately different environments, which is a setup favorable to MEDC. The smart contract analysis reports gas costs but no actual deployment or security evaluation. The heavy reliance on the authors' own earlier papers for the environment and the single-expert version is not a flaw by itself, but it does mean the incremental contribution here is modest.\n\nOverall, the paper is a reasonable systems-and-empirics contribution that would benefit from major revision rather than a desk reject. The method is not ready to cite as a principled PPO variant until the on-policy issue is resolved and the results are reproduced with proper variance reporting.\n\nI would accept it for peer review with the expectation of substantial revision. A meticulous referee should be asked to verify the distribution-shift concern and require ablations and seed variance.","headline":"The multi-expert action-suggestion idea is plausible and the experiments are consistent, but the paper's on-policy justification is wrong and the evidence lacks seeds, error bars, and code.","tokens_in":19944,"tokens_out":1507,"would_cite":false,"duration_ms":19278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A probability-filtered expert-suggestion rule lets multi-agent PPO learn faster under sparse rewards while staying resilient to faulty and malicious shared models.","keywords":["Multi-Agent Deep Reinforcement Learning","Demonstration Cloning","Proximal Policy Optimization","Sparse Rewards","Federated Reinforcement Learning","Blockchain","Smart Contracts","Imitation Learning"],"falsifier":"Record the importance ratios $r_t(\\theta)$ from Equation (2) during MEDC episodes and check whether expert-suggested actions routinely push $r_t$ outside $[1-\\varepsilon, 1+\\varepsilon]$; if clipping activates mainly on expert-influenced steps, the reported speed-ups could stem from biased updates rather than better exploration. A cleaner control is to replace the expert suggestions in the same training schedule with actions sampled from the agent's own policy and see whether the learning-speed advantage disappears.","tokens_in":18960,"feed_emoji":"🤖","tokens_out":10514,"duration_ms":91120,"temperature":0.7,"pith_summary":"The paper proposes MEDC, a method that lets previously trained expert models suggest actions to agents during Proximal Policy Optimization training, but accepts a suggestion only when the learner's own policy assigns that action a probability above a threshold $Q$. The claim is that this accelerates learning under sparse rewards compared with plain multi-agent reinforcement learning, and that it outperforms federated reinforcement learning, reward shaping, and imitation-learning-assisted reinforcement learning while staying resilient to faulty and malicious experts. Because agents still learn from the rewards they collect after executing expert-suggested actions, bad suggestions do not poison the policy; at worst the method matches a sparse-reward baseline. A companion blockchain-based model-sharing framework with smart contracts allocates shared models to requesters by reputation and environmental similarity.","feed_headline":"Filtered expert actions speed sparse-reward multi-agent RL","feed_subtitle":"Blockchain-shared experts guide training; the Q-threshold keeps PPO stable and tolerates faulty models.","key_machinery":"The load-bearing mechanism is the $Q$-threshold acceptance rule in Algorithm 2: given the agent's policy distribution $P$ over actions, an expert-suggested action $a_e$ is executed only if $P(a_e) > Q$, and the agent then takes the highest-probability surviving suggestion. This converts expert knowledge into exploration guidance while staying within the on-policy PPO framework, and it self-attenuates as the policy sharpens. The model-sharing side is carried by a smart-contract allocation mechanism that ranks shared models by the quality metric $\\mathrm{QoS}_m = (\\mathrm{Rep}_i \\times \\mathrm{Rep}_m) / D_m$, where $\\mathrm{Rep}_i$ is the owner's reputation, $\\mathrm{Rep}_m$ is the model's reputation, and $D_m$ is a weighted distance between the model's training environment and the requester's environment.","core_discovery":"The central discovery is that an expert's action suggestion can serve as a safe exploration hint for an on-policy learner if it passes a probability filter under the agent's current policy. By following only suggestions with $P(a_e) > Q$ and then learning from the rewards those actions actually produce, MEDC keeps the PPO update nominally on-policy while making sparse rewards reachable early in training. The threshold automatically reduces expert influence as the policy becomes confident, since fewer expert actions will clear $Q$. The paper reports that across target localization, fleet coordination, and maze cleaning, MEDC learns faster than federated averaging, reward shaping, and behavioral-cloning-assisted RL, and in the presence of faulty or malicious experts its performance degrades at most to the no-expert sparse-reward case.","pith_inferences":["The fixed threshold $Q=0.05$ is a hyperparameter; an adaptive $Q$ schedule that starts low and rises with policy confidence could extract more early guidance while further reducing bias later in training.","The acceptance rule is discrete-action-specific (argmax over surviving actions); for continuous action spaces it would need a density- or divergence-based filter, which is not addressed.","If the on-policy bias is material, a variant with off-policy correction or importance sampling for expert-suggested actions would make the speed-up claim robust to the distribution-shift objection.","The QoS allocation depends on the environment-attribute distance $D_m$ with application-specific weights, so applying the framework to a new problem requires defining those attributes and weights."],"forward_implications":["Different neural-network architectures can be reused as experts, since no weight averaging or distillation is needed.","Simple sparse rewards become viable, avoiding the engineering cost and local-optima risk of shaped reward functions.","Expert influence shrinks automatically over training, since a confident policy assigns low probability to suggested actions and they fail the $Q$ filter.","Faulty, random, biased, or malicious experts do not degrade learning below the plain sparse-reward baseline in the tested settings.","The blockchain-based sharing and allocation provides traceable, autonomous model exchange without a single trusted server, at gas costs the paper reports as low."],"supporting_citations":[{"why":"It supplies the clipped surrogate objective (Eq. 1) that MEDC modifies and uses as the base PPO learner.","marker":"[34]"},{"why":"It provides Generalized Advantage Estimation, used to compute the advantage term in the PPO update.","marker":"[35]"},{"why":"It defines the target-localization environment and the CNN policy architecture used in the main experiments.","marker":"[7]"},{"why":"It supports the paper's explanation of why federated averaging struggles when environments differ, motivating the MEDC comparison.","marker":"[17]"},{"why":"It is one of the federated reinforcement learning baselines MEDC is benchmarked against.","marker":"[25]"},{"why":"It is another federated reinforcement learning baseline used in the learning-speed comparison.","marker":"[26]"},{"why":"It is an imitation-learning-assisted multi-agent RL baseline that MEDC outperforms.","marker":"[31]"},{"why":"It is a further imitation-learning-assisted multi-agent RL baseline that MEDC outperforms.","marker":"[32]"}],"fun_headline_variants":["Expert hints pass policy filter to boost multi-agent RL","Blockchain shares expert models to accelerate multi-agent RL","Policy-filtered demonstrations speed sparse-reward MARL","Safe expert actions cut training time for multi-agent RL","Faulty experts tolerated via filtered demonstration cloning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that accepting an expert-chosen action whenever the agent's own policy assigns it probability above $Q$ keeps the PPO updates valid, even though that action was not sampled from the agent's policy and therefore makes the clipped objective in Equation (1) a biased estimate of the policy gradient.","fun_headline_variants_meta":{"raw":{"variants":["Expert hints pass policy filter to boost multi-agent RL","Blockchain shares expert models to accelerate multi-agent RL","Policy-filtered demonstrations speed sparse-reward MARL","Safe expert actions cut training time for multi-agent RL","Faulty experts tolerated via filtered demonstration cloning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1327,"prompt_tokens":949,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":303}},"tokens_in":565,"tokens_out":378,"duration_ms":4552,"temperature":1.0,"reasoning_tokens":303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:48:54.750306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the importance ratios $r_t(\\theta)$ from Equation (2) during MEDC episodes and check whether expert-suggested actions routinely push $r_t$ outside $[1-\\varepsilon, 1+\\varepsilon]$; if clipping activates mainly on expert-influenced steps, the reported speed-ups could stem from biased updates rather than better exploration. A cleaner control is to replace the expert suggestions in the same training schedule with actions sampled from the agent's own policy and see whether the learning-speed advantage disappears.","supporting_citations":[{"cited_title":"High- dimensional continuous control using generalized advantage estimation,","cited_arxiv_id":null,"evidence_quote":"It provides Generalized Advantage Estimation, used to compute the advantage term in the PPO update."},{"cited_title":"Federated transfer reinforcement learning for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"It is another federated reinforcement learning baseline used in the learning-speed comparison."},{"cited_title":"Primal: Pathfinding via reinforcement and imitation multi- agent learning,","cited_arxiv_id":null,"evidence_quote":"It is an imitation-learning-assisted multi-agent RL baseline that MEDC outperforms."},{"cited_title":"Primal 2: Pathfind- ing via reinforcement and imitation multi-agent learning-lifelong,","cited_arxiv_id":null,"evidence_quote":"It is a further imitation-learning-assisted multi-agent RL baseline that MEDC outperforms."}],"review_version":1}