{"id":"efb096ed-7c0a-4070-ad4b-f9c986b3f7a1","arxiv_id":"2502.06113","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"PSO-guided exploration is added to MASAC and claimed to reduce training time for a 3-agent coverage planning task, without quantitative validation.","lead":"This paper tests whether adding a particle swarm optimizer to the exploration step of a multi-agent reinforcement learning algorithm speeds up training for an underwater coverage mission. The authors report faster learning in a simulated 2D area-coverage task, but the evidence is qualitative and no code or data are provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that PSO-guided exploration reduces training time is unsupported because the wall-clock cost of the PSO oracle is never measured, so fewer environment interactions may not translate into less real training time.","rationale":"The reader's weakest-assumption analysis correctly identifies that the acceleration claim conflates sample efficiency with wall-clock training time. My independent read of Sections 3.1, 3.2, and 4 confirms that no runtime measurement is reported, no PSO solution-quality check is performed, and no quantitative comparison is given. The absence of code and seeds makes reproduction impossible, but the more specific failure is that the paper's own headline quantity—training time—is never measured. The PSO configuration (50 particles, 100 iterations) is substantial enough that a careful reviewer should not assume the oracle call is free. I agree with the REJECT verdict: the central claim is not supported by the demonstrated evidence, and the missing wall-clock comparison is a concrete, testable gap rather than a preference about presentation.","tokens_in":9486,"tokens_out":1477,"duration_ms":17578,"concrete_test":"Instrument the training loop on identical hardware and report wall-clock time per training step for the PSO branch (50 particles, 100 iterations) versus the epsilon-greedy branch, then compute total wall-clock time to reach a fixed mean-reward threshold with and without PSO. If the PSO overhead is larger than the interaction savings, the training-time claim is refuted. As a secondary check, record the fitness of the best PSO action against an exhaustive one-step action sweep to quantify whether the heuristic is actually near-optimal.","verdict_should_be":"REJECT","load_bearing_attack":"The paper claims in Section 4 that 'the addition of heuristics within the exploration strategy leads to lower training time,' and the abstract repeats that 'the resulting agent requires fewer interactions to reach optimal performance.' The only evidence offered is qualitative: plots showing higher mean rewards with PSO than without. The load-bearing assumption is that the PSO heuristic is cheap enough that the reduction in environment interactions dominates total training cost. This is never tested. Section 3.1 states that instead of random actions, the PSO solution is used in exploration, and Section 3.2 gives PSO parameters of 50 particles and 100 iterations. If each such call requires 100 iterations of swarm updates per exploration step, the computational overhead can easily exceed the savings in environment steps, especially because PSO is an optimization procedure run inside the learning loop. The paper also never verifies that PSO actually returns near-optimal actions for the stated fitness function. Since the fitness function is identical to the reward function and the environment is fully observable in simulation, PSO could be acting as an expensive local planner; that would explain higher reward but would not establish a practical training-time reduction. The confounding is not merely a missing error bar: the headline quantity is 'training time,' and the paper measures only episodes to convergence, not seconds or FLOPs. Even granting better sample efficiency, the central claim may be false in the metric the authors themselves name.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the coordination of a multi-agent system (MAS) of autonomous underwater vehicles (AUVs) for coverage planning and detection of objects of interest. It proposes combining Multi-Agent Soft Actor-Critic (MASAC) with Particle Swarm Optimization (PSO) as a biologically inspired heuristic: during epsilon-greedy exploration, the random action is replaced by a PSO-output action that maximizes a fitness function set equal to the MARL reward. The authors report preliminary results on a 2D coverage task with three agents and claim that the method reduces training time by requiring fewer environment interactions.","tokens_in":9756,"tokens_out":3728,"duration_ms":36102,"significance":"The problem is practically important: MARL methods for underwater and space multi-agent missions suffer from long training times, and a method that reliably accelerates learning without sacrificing performance would be valuable. The paper also contributes a customizable coverage environment and a clear integration of PSO into the epsilon-greedy scheme. However, the significance as demonstrated is limited: the central claim of reduced training time rests on qualitative plots with no quantitative support, and the PSO's computational overhead is not measured. The design choice of using the reward function as the PSO fitness function makes the 'heuristic' effectively an online optimizer of the target objective, which weakens the claim that bio-inspired heuristics per se improve MARL. If the authors can provide rigorous evidence and an honest accounting of computational cost, the idea could be a useful step toward reducing MARL training time in simulation-to-real settings.","major_comments":[{"comment":"The central claim that 'the addition of heuristics within the exploration strategy leads to lower training time' is not supported by the presented evidence. The results are two qualitative learning curves with no numerical reward values, no error bars, no number of independent random seeds, and no definition of what constitutes convergence or 'training time.' Without these, the reader cannot assess the variance of the results or verify that the observed improvement is statistically meaningful.","section":"Section 4, Figures 2 and 3"},{"comment":"The paper equates fewer environment interactions with lower training time, but it never measures wall-clock training time or the computational cost of the PSO. Each PSO evaluation uses 50 particles and 100 iterations (Section 3.2), and if invoked on exploration steps, this overhead can easily dominate the savings in environment steps. The claim of 'lower training time' is thus unsubstantiated; the correct metric would be total computation time or a cost model that includes PSO's internal optimization.","section":"Section 3.2 and Section 4"},{"comment":"The PSO fitness function is defined to be identical to the MARL reward function. This means the 'heuristic' is not an independent source of prior knowledge but an online maximizer of the same objective the agent is learning. The observed superiority of PSO-guided exploration over random exploration is therefore unsurprising and confounded: the comparison effectively pits RL exploration against a reward-based planner. To support the claimed value of bio-inspired heuristics, the authors should compare against other action-selection methods (e.g., a random action with equal compute, a separate heuristic not derived from the reward, or a planning baseline) and discuss the distinction between a heuristic and an optimizer.","section":"Section 3.1"},{"comment":"The experimental setup is insufficiently specified for reproducibility. Missing details include MASAC hyperparameters (learning rates, network sizes, batch size), the replay buffer size, the epsilon schedule, the number of training episodes, and the precise integration of PSO (how often it is called, whether it is applied to all agents or a subset, and how its output is combined with the epsilon-greedy probability). These details are necessary for the results to be independently verified.","section":"Section 3 (overall)"}],"minor_comments":[{"comment":"The reward function equation is garbled with non-ASCII characters and is not legible; it should be typeset properly using LaTeX math notation.","section":"Section 3.2, reward function"},{"comment":"The figures lack axis labels, units, legends, and a description of what the curves represent (e.g., mean reward per episode over how many runs).","section":"Figures 2 and 3"},{"comment":"The conclusion says the method 'reduces the training time,' while Section 4 only claims 'these preliminary results support the hypothesis.' The language in the abstract and conclusion should be aligned with the level of evidence actually presented.","section":"Section 5, Conclusion"},{"comment":"The related work on Heuristically Accelerated Reinforcement Learning (HARL) is cited but not discussed in enough depth; the novelty of the proposed PSO-as-heuristic approach relative to existing HARL methods should be stated explicitly.","section":"Section 2.2"},{"comment":"There are several incomplete references (e.g., [39] has no year or venue, [41] is missing page numbers) and inconsistent formatting throughout. The reference list should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads as an early workshop result. The idea is worth pursuing, but the current evidence is far below the standard for a journal publication: no quantitative numbers, no seeds, no wall-clock time, and a confounded heuristic. The authors need to add substantial experiments and, more importantly, reconsider the design choice of using the reward function as the PSO fitness function if they want to claim a contribution to bio-inspired heuristics. I would welcome a revision that addresses these points, but the paper as it stands is not publishable. I suggest the editor invite a thorough revision rather than a desk reject, as the underlying problem is relevant and the proposed direction is not obviously wrong."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a straightforward but legitimate extension of HARL — replace epsilon-greedy random actions with PSO-guided actions in MASAC, for a 2D coverage task. It is not a groundbreaking idea, but it is a sensible combination that I do not think is in the literature, and the paper is refreshingly clear about its own limits. The problem is that the central claim — \"reduces training time\" — is not supported by the evidence they present.\n\nWhat is good: The related work on HARL and bio-inspired methods is competent and appropriately cites Bianchi et al. and the HARL line. The environment is described in enough detail to be reimplemented (3 agents, 30x30 grid, 60-degree forward sensor, reward function). The authors are explicitly honest that these are \"preliminary results\" and they list sensible future work. The idea of injecting domain knowledge through PSO into the exploration policy is a reasonable research direction.\n\nWhat is soft: The results section shows only two qualitative learning curves, no numbers, no error bars, no seed counts. More importantly, the headline metric is training time, but they only report environment interactions (episodes or steps). They never measure wall-clock time or compute the PSO overhead. Section 3.1 gives 50 particles and 100 iterations per PSO call; that is a lot of computation inside the learning loop, and it could easily outweigh the savings in environment steps. The stress-test note is right: \"fewer interactions\" is not \"lower training time.\" On top of that, the PSO fitness function is identical to the reward function, so PSO is effectively a local planner optimizing the same objective. That is not circular in a logical sense, but it makes the acceleration unsurprising and raises a fairness question about whether you are comparing RL to RL plus a planner.\n\nI agree with the reader's verdict: reject as-is. But it is not a worthless paper. The kernel is worth pursuing, and with wall-clock time measured, PSO cost reported, and multiple seeds with variance, it could become a decent conference paper. The citation pattern looks fine; the self-citations are relevant and the HARL citations are proper.\n\nWho is it for? People working on HARL, multi-agent coverage, or sim-to-real for AUVs might skim it. I would send it to review if I were the editor, because the idea is reasonable and the authors are clearly able to respond to feedback. It just needs substantially more evidence.\n\nMy recommendation: if you are reviewing it, reject with clear, constructive comments focusing on the missing wall-clock measurement and the reward-function confounding. If you are the editor, send it out — a good referee can push it toward something publishable.","headline":"A clear, honest HARL extension whose central training-time claim is unsupported because the paper never measures wall-clock cost.","tokens_in":10298,"tokens_out":2124,"would_cite":false,"duration_ms":18608,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding Particle Swarm Optimization to the epsilon-greedy exploration of Multi-Agent Soft Actor-Critic lets three simulated underwater vehicles learn a 2D coverage task with fewer environment interactions, which it…","keywords":["coverage planning","multi-agent reinforcement learning","Particle Swarm Optimization","heuristically accelerated reinforcement learning","MASAC","autonomous underwater vehicles","exploration strategy"],"falsifier":"A direct falsifier is a wall-clock comparison: run the same 2D coverage task with and without PSO, counting not just episodes but total computation time including PSO's 100 iterations over 50 particles for every exploration step; if the PSO-trained agent reaches the same coverage in fewer episodes but not in less time, the central claim of reduced training time fails. A second check is to compare PSO's proposed actions with actions found by a fine exhaustive search over a small action grid, since the proposed mechanism requires those proposals to actually be high-reward.","tokens_in":9257,"feed_emoji":"🌊","tokens_out":7041,"duration_ms":56446,"temperature":0.7,"pith_summary":"The paper tries to establish that swapping random exploration for Particle Swarm Optimization (PSO) proposals inside Multi-Agent Soft Actor-Critic (MASAC) shortens the learning phase of a multi-agent coverage task. The motivation is that MARL's long training times confine underwater autonomy to simulation, so anything that reduces the number of environment interactions could move these systems closer to real deployment. In a customizable 2D coverage environment with three agents, the authors report that PSO-guided epsilon-greedy exploration reaches higher mean rewards earlier in training than the same algorithm with random exploration. The paper presents these as preliminary results supporting the hypothesis that heuristics in the exploration strategy lead to lower training time.","feed_headline":"PSO-guided exploration cuts multi-agent coverage training time","feed_subtitle":"Three simulated underwater robots get better coverage in fewer episodes when PSO, not noise, drives exploration.","key_machinery":"The central object is a PSO-guided epsilon-greedy exploration rule. Whenever the policy chooses to explore, instead of sampling a uniform random action, the agent executes the best candidate found by PSO, which in the reported runs uses 50 particles and 100 iterations to maximize a fitness function identical to the RL reward function. This mechanism is attached to MASAC, a multi-agent variant of the Soft Actor-Critic actor-critic algorithm with stochastic policies and a replay buffer; the intended effect is to bias exploration toward high-quality regions of the action space from the first episode, sharpening the exploitation-exploration trade-off and accelerating policy-gradient updates.","core_discovery":"The central claim is that heuristically accelerated exploration, specifically PSO embedded in epsilon-greedy action selection, lowers the training time of MASAC on a continuous 2D coverage task. Because the PSO fitness function is defined to be the same as the MARL reward function, the actions PSO proposes are associated with high rewards from the very beginning of training. Since policy-gradient updates work by comparing actions through the advantage function, feeding the agent better-than-average actions early lets it improve its return immediately rather than waiting to stumble on such actions by chance. The paper states this as: \"These preliminary results support the hypothesis that the addition of heuristics within the exploration strategy leads to lower training time.\"","pith_inferences":["The paper counts episodes, not wall-clock time, so the logical next test is to measure total computation time including PSO's 100 iterations over 50 particles at every exploration step; the headline claim stands or falls on that comparison.","The reported acceleration was measured with three homogeneous agents on a 30-by-30 grid; a natural extension is to check how the advantage behaves as the number of agents grows, since PSO's search space expands with the joint action dimension.","PSO's own hyperparameters (particles, iterations, inertia, cognitive and social coefficients) are part of the training cost, so an adaptive or learned PSO configuration might amplify the acceleration or erase it entirely.","The paper itself notes that switching to the automatic-entropy version of MASAC creates stability challenges; a fair extension would test whether PSO-guided exploration also stabilizes that variant."],"forward_implications":["MASAC should reach the same coverage performance in fewer episodes when exploration is guided by PSO, reducing the number of simulator interactions needed before the policy is usable.","Because the PSO fitness is the RL reward itself, the same insertion can in principle be applied to any epsilon-greedy MARL algorithm without changing the reward design.","Providing high-reward actions early makes each policy-gradient update more informative, since advantage estimates are computed relative to better-than-average actions.","If the interaction savings survive the overhead of running PSO, the approach is a step toward learning policies on-board real autonomous underwater vehicles, where each interaction costs battery and mission time."],"supporting_citations":[{"why":"Supplies the Particle Swarm Optimization algorithm that the paper inserts into epsilon-greedy exploration.","marker":"[36]"},{"why":"Provides the modified PSO variant and parameter values (inertia 0.8, cognitive and social components 2.0) used in the experiments.","marker":"[37]"},{"why":"Defines Soft Actor-Critic, the base algorithm whose multi-agent version MASAC is built on.","marker":"[20]"},{"why":"Provides the epsilon-greedy stochastic exploration strategy that the paper modifies by replacing random actions with PSO proposals.","marker":"[25]"},{"why":"Establishes the Heuristically Accelerated Reinforcement Learning (HARL) framing that the proposed PSO-guided exploration belongs to.","marker":"[28]"},{"why":"Describes the experience replay mechanism that stores the PSO-guided interactions and is invoked in the paper's discussion of future work.","marker":"[40]"}],"fun_headline_variants":["PSO-guided exploration speeds up underwater multi-agent training","Bio-inspired heuristics cut training time for underwater robot swarms","Faster MARL training with PSO-based exploration for coverage tasks","Swarm intelligence accelerates multi-agent reinforcement learning","PSO heuristic reduces training episodes for underwater coverage agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The acceleration claim assumes that PSO's proposed actions are cheap enough and good enough that the total training time really drops, but the paper neither measures PSO's computational cost nor verifies that its proposals are near-optimal.","fun_headline_variants_meta":{"raw":{"variants":["PSO-guided exploration speeds up underwater multi-agent training","Bio-inspired heuristics cut training time for underwater robot swarms","Faster MARL training with PSO-based exploration for coverage tasks","Swarm intelligence accelerates multi-agent reinforcement learning","PSO heuristic reduces training episodes for underwater coverage agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1247,"prompt_tokens":969,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":199}},"tokens_in":585,"tokens_out":278,"duration_ms":2828,"temperature":1.0,"reasoning_tokens":199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:42:22.334052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier is a wall-clock comparison: run the same 2D coverage task with and without PSO, counting not just episodes but total computation time including PSO's 100 iterations over 50 particles for every exploration step; if the PSO-trained agent reaches the same coverage in fewer episodes but not in less time, the central claim of reduced training time fails. A second check is to compare PSO's proposed actions with actions found by a fine exhaustive search over a small action grid, since the proposed mechanism requires those proposals to actually be high-reward.","supporting_citations":[{"cited_title":"Value-based methods [22] estimate the value of each state or state -action pair to derive optimal policies by selecting actions that maximize the cumulative reward","cited_arxiv_id":null,"evidence_quote":"Defines Soft Actor-Critic, the base algorithm whose multi-agent version MASAC is built on."},{"cited_title":"Ant system: optimization by a colony of cooperating agents,","cited_arxiv_id":null,"evidence_quote":"Supplies the Particle Swarm Optimization algorithm that the paper inserts into epsilon-greedy exploration."},{"cited_title":"The bees algorithm and mechanical design optimisation,","cited_arxiv_id":null,"evidence_quote":"Provides the modified PSO variant and parameter values (inertia 0.8, cognitive and social components 2.0) used in the experiments."},{"cited_title":"Entropy regularized actor-critic based multi-agent deep reinforcement learning for stochastic games,","cited_arxiv_id":null,"evidence_quote":"Provides the epsilon-greedy stochastic exploration strategy that the paper modifies by replacing random actions with PSO proposals."},{"cited_title":"Leveraging world model disentanglement in value-based multi- agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Establishes the Heuristically Accelerated Reinforcement Learning (HARL) framing that the proposed PSO-guided exploration belongs to."},{"cited_title":"Group search optimizer: An optimization algorithm inspired by animal searching behavior,","cited_arxiv_id":null,"evidence_quote":"Describes the experience replay mechanism that stores the PSO-guided interactions and is invoked in the paper's discussion of future work."}],"review_version":1}