{"id":"b32dd549-d636-479c-a760-f9e7f77c410f","arxiv_id":"2412.07617","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An ensemble of behavior-cloned policies trained with a hidden-activation alignment penalty produces more consistent actions and higher mean episode returns than standard ensemble behavior cloning.","lead":"Swarm Behavior Cloning adds a small penalty that keeps the internal representations of an ensemble of behavior-cloning agents similar. In eight simulated control tasks, this reduces disagreement among the agents and improves average scores, especially in harder environments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical claim that Swarm BC improves returns is not statistically secured: only 5 seeds, no significance tests, and hyperparameters tuned on one environment, so the reported gains may be noise.","rationale":"The reader's weakest_assumption is the theoretical bridge in Section 7, which is indeed a genuine flaw: the proof only shows concentration of the product density p(h_k)^N and never derives loss (2) from it. However, that flaw does not bear on whether the proposed loss empirically improves performance; it only affects the stated mechanism. The central claim, as framed in the strongest_claim, is empirical: the loss reduces action difference and improves return. The load-bearing condition for that claim is the reliability of the experimental evidence. With only 5 seeds, no significance tests, and hyperparameters tuned on one environment, the reported improvements could be due to chance or to favorable choices that do not transfer. The reader also requested statistical reporting as a revision, so my concern is partially aligned with their rationale. Since the reader's verdict is already CONDITIONAL and includes the need for significance testing, my concern does not move the verdict; it reinforces the need for those revisions. I chose a concrete test on the flagship HalfCheetah result because it is the most striking claim and the easiest to re-check statistically.","tokens_in":12115,"tokens_out":11414,"duration_ms":103822,"concrete_test":"Re-run the HalfCheetah 8-expert-episode condition with 30 independent seeds, using loss (2) with τ=0.25 and N=4 and the same training protocol as the paper. Compute the mean and 95% confidence interval of the scaled return for Swarm BC and Ensemble BC, and run a two-sample t-test as well as a Mann-Whitney U test. If the p-value exceeds 0.05 or the confidence intervals overlap substantially, the flagship result is not statistically established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Swarm BC improves mean episode return and reduces action difference over BC and Ensemble BC. The truth of this claim rests on the experiments in Section 6, not on the theoretical analysis in Section 7. Those experiments use 5 seeds per condition, and the word 'significantly' appears without any significance test, confidence interval, or effect size. The hyperparameters τ=0.25 and N=4 are selected from an ablation on Walker2D alone (Figure 5) and then fixed for all other environments; whether these values are sensible elsewhere is unknown. The shaded areas in Figure 3 are not described as standard errors across seeds, and with only 5 seeds it is plausible that Swarm BC and Ensemble BC overlap in several environments. If, for example, the HalfCheetah 8-episode gap (0.72 vs 0.17) carries a standard deviation of 0.5, the difference would not be statistically meaningful. Because the empirical improvement is the basis for the paper's conclusion, the absence of statistical validation is the most load-bearing weakness. The theory gap identified by the reader is real, but it concerns the explanation of why the method works, not whether it works.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Swarm Behavior Cloning (Swarm BC), a variant of ensemble behavior cloning in which N policies are trained jointly with an additional regularizer that penalizes pairwise differences between their hidden feature activations. The objective is Eq. (2): the standard per-policy MSE plus tau times the sum over layers and policy pairs of squared hidden-feature differences. The ensemble action is the average of the N predicted actions. The authors evaluate the method on eight OpenAI Gym environments, comparing scaled episode returns and mean action differences against BC and Ensemble BC, and include ablations over tau and N. Section 7 presents a theoretical analysis intended to show that the regularized ensemble concentrates on the mode of the hidden-activation distribution.","tokens_in":12363,"tokens_out":6277,"duration_ms":58459,"significance":"If the empirical claim holds, Swarm BC is a simple, fully offline regularizer for ensemble imitation learning that requires no additional environment interaction, and the paper would provide a useful baseline for a known failure mode of ensemble BC. The evaluation is not circular: it uses external benchmark environments and independent Stable-Baselines3 expert policies, and the paper explicitly reports ablations for both introduced hyperparameters. However, the load-bearing empirical evidence is currently weak (five seeds, no significance tests or confidence intervals), and the theoretical analysis in Section 7 does not bridge the actual loss in Eq. (2) to the mode-concentration argument it proves. The central idea is plausible and worth further scrutiny, but the manuscript in its present form does not convincingly establish the claimed improvements.","major_comments":[{"comment":"The central empirical claim that Swarm BC significantly improves episode returns is not statistically supported. The paper reports means over 5 seeds and 20 evaluation episodes, but never reports per-seed values, confidence intervals, effect sizes, or a significance test; the word 'significantly' appears repeatedly without a test. With n=5, the HalfCheetah contrast (0.72 vs 0.17 scaled return at 8 episodes) is not interpretable without knowing seed-level variance, and the shaded 'standard deviation' bands in Figure 3 are not a substitute for a paired comparison. Please provide per-seed results and appropriate significance tests or bootstrap confidence intervals, or soften the conclusions.","section":"Section 6, Figure 3"},{"comment":"Hyperparameter selection undermines the generality claim. The values tau=0.25 and N=4 are chosen from the Walker2D ablation (Figure 5) and then fixed for all eight environments, yet the method's behavior depends directly on the strength tau of the regularizer. The ablation itself also uses 'significantly' without a statistical test. Please add a sensitivity analysis on more than one environment, or demonstrate that the reported ranking is stable across reasonable tau values.","section":"Section 6, Figure 5"},{"comment":"The theoretical section does not prove what it claims. The text states that 'training an ensemble with similar feature activations corresponds to finding the global mode' of p(h_k), but Section 7 only proves concentration of the product density p(h_k)^N; it never derives this product form from the actual training objective in Eq. (2), nor does it show that SGD on Eq. (2) samples from p_N(h_k). Moreover, the Proposition is stated in the limit tau -> 0 and N -> infinity, whereas the experiments use finite tau=0.25 and N=4. Please supply a derivation connecting Eq. (2) to the mode-seeking argument, or explicitly label Section 7 as intuition and remove the claim that it is a proof.","section":"Section 7"},{"comment":"The reduction in mean action difference is quantified for only four of the eight environments (Figure 4), and the only numerical reductions given (44% for BipedalWalker, 11% for Ant) are not accompanied by errors or tests. The paper's second hypothesis is therefore only partially verified. Please report aggregate reductions and variability across all environments, or restrict the claim accordingly.","section":"Section 6, Figure 4"}],"minor_comments":[{"comment":"The final expression for N uses p(phi+)/p(phi#) where the densities at stake are p(h+_k)/p(h#_k); the notation should be consistent, and 'deviation' should be 'derivative'.","section":"Section 7, final display"},{"comment":"The text says datasets D contain x in [1,8] episodes, but the x-axes in Figure 3 begin at different values for different environments (e.g., HalfCheetah at 3, CartPole at 1); please clarify the exact protocol used for each environment.","section":"Section 6, experimental protocol"},{"comment":"The definition is numbered 'Definition 1' but referred to later as 'Definition 3.1'; use consistent numbering.","section":"Section 3, Definition 1"},{"comment":"The Input line should use set notation D = {(s, a = pi_e(s))} rather than D = (s, a = pi_e(s)), and the phrase 'in Chapter 5' should be 'in Section 5'.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the core idea is simple and potentially useful, but the current empirical protocol is too weak to justify the strong wording 'significant improvements,' and the theory section as written overclaims by presenting a concentration result as if it followed from loss (2). These are fixable with additional experiments and a repositioned or properly derived theoretical section, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes a straightforward modification to ensemble behavior cloning: penalize the L2 distance between hidden activations of ensemble members, with scale tau. That loss is not in the cited prior work, and the idea is honestly framed as a fully offline version of Brantley et al.'s disagreement-regularized IL. The regularizer is cheap and the intuition—reduce action scatter in underrepresented states—is reasonable. The reported reductions in mean action difference are consistent across environments, and the gains in scaled return, e.g., HalfCheetah 0.72 vs 0.17, are large enough to take seriously even with noise.\n\nThe soft spots are real, and the main one is statistical. Five seeds per condition, no significance tests or confidence intervals, and the word 'significantly' appears without any backing. With that few seeds, several of the gaps in Figure 3 could easily overlap. The hyperparameters tau=0.25 and N=4 come from an ablation on Walker2D alone and are then fixed everywhere; there is no analysis of how sensitive other environments are to those choices. I also could not find any released code or hyperparameter settings, which would be a minimal requirement for a methods paper.\n\nThe theory section is a separate problem. It sets up p(h_k), then proves that the product density p(h_k)^N concentrates on the mode as N grows. That is a true but weak concentration statement, and it never derives the actual loss (2). The paper says 'we now show that training an ensemble with similar feature activations corresponds to finding the global mode,' but the proof never makes contact with the regularization term. The reader's weakest assumption is exactly right: if that bridge fails, the theory is just a disconnected observation. My recommendation would be to either remove the theory or rewrite it so that it starts from loss (2) and shows what it actually regularizes.\n\nOverall, the empirical result is plausible and useful enough to deserve reviewer time, but the paper is not there yet. I would send it to peer review with a request for more seeds, significance testing, released code, and a repaired or removed theory section. If those changes land, I would cite it as a simple regularizer for ensemble IL.","headline":"A simple, plausible regularizer for ensemble BC; the empirical claim is suggestive but not statistically secured, and the theory doesn't prove what it claims.","tokens_in":12865,"tokens_out":2466,"would_cite":true,"duration_ms":21174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Penalizing hidden-feature differences among ensemble policies reduces action divergence and improves imitation returns in eight control environments.","keywords":["Swarm Behavior Cloning","Behavior Cloning","Imitation Learning","Ensemble learning","Action difference","Hidden feature regularization","Mean episode return","Reinforcement Learning"],"falsifier":"Estimate $p(h_k)$ for a fixed state $s$ by training many independent BC policies on the same dataset and histogramming one-layer hidden activations; train a Swarm BC ensemble with that same setting and compare its converged hidden activations to the estimated global mode. If the Swarm activations do not track the mode as $\\tau$, $N$, and the number of estimation samples grow, the paper's theoretical equivalence between loss (2) and mode-seeking is false. A simpler empirical falsifier: on any of the eight environments, set $\\tau$ so large that the BC term is negligible; the proposition predicts the ensemble still finds the mode of $p(h_k)^N$, whereas in practice the policies would just collapse to identical weights, which can be checked directly.","tokens_in":11931,"feed_emoji":"🐝","tokens_out":6435,"duration_ms":52863,"temperature":0.7,"pith_summary":"Behavior cloning with an ensemble of neural policies suffers from a failure mode the paper calls increasing action differences: in states underrepresented in the expert data, the policies disagree and their averaged action can land in low-value regions. Swarm Behavior Cloning changes the training loss so that policies are not only trained to match expert actions but also penalized for having dissimilar hidden-feature activations, encouraging the ensemble to behave like a coherent swarm. Across eight benchmark control environments the method reduces mean action difference and improves mean episode return relative to both single-policy BC and standard Ensemble BC, with the largest gains in high-dimensional environments such as HalfCheetah. The paper also argues that this regularization concentrates the ensemble on the mode of the hidden-activation distribution, giving a theoretical justification for why forcing agreement helps.","feed_headline":"Swarm Behavior Cloning cuts action spread, boosts returns","feed_subtitle":"Penalizing hidden-feature disagreement among cloned policies lifts scaled returns from 0.17 to 0.72 in HalfCheetah.","key_machinery":"The central object is the coupled ensemble loss (equation 2): the standard per-policy behavior-cloning term plus a pairwise penalty $\\tau \\sum_{i<j} \\|h_i^k(s) - h_j^k(s)\\|^2$ at each hidden layer $k$, for a user-chosen coefficient $\\tau$. This term is what turns independently trained policies into a swarm: it pushes the internal feature representations toward one another while leaving each policy's output head free, preserving some diversity in computation. The theoretical complement is a density-concentration argument: the distribution of $N$ independently sampled hidden activations $h_k$ is proportional to $p(h_k)^N$, whose mass shifts to the global mode as $N$ grows, and the paper claims that optimizing loss (2) realizes this concentration.","core_discovery":"On the paper's own terms, Swarm Behavior Cloning establishes that a simple additive regularizer on hidden-layer activations of an ensemble of behavior-cloned policies reduces the mean action difference between members and improves the quality of the averaged action. The combined loss is $L(s,a) = \\sum_i (\\pi_i(s)-a)^2 + \\tau \\sum_k \\sum_{i<j} \\|h_i^k(s)-h_j^k(s)\\|^2$, where the second term couples all pairs of policies at every hidden layer. With $\\tau = 0.25$ and $N = 4$ policies, the method matches or beats both BC and Ensemble BC in all eight tested environments; for HalfCheetah with 8 expert episodes the scaled return is 0.72 versus 0.17 for Ensemble BC. The paper additionally claims that training such a coupled ensemble approximates sampling from $p_N(h_k) \\propto p(h_k)^N$, so that as $N$ grows the shared features concentrate on the global mode of the single-policy activation density.","pith_inferences":["If the mode-concentration argument is right, the same hidden-feature coupling should also suppress action variance in out-of-distribution states that never appear in the training data, a point the experiments do not directly test.","Because the regularizer only couples features, it could in principle be combined with diversity-promoting terms to prevent all policies from collapsing to identical networks, though the paper does not explore that.","A direct test of the theory would be to estimate $p(h_k)$ empirically from a large pool of independent BC policies and check whether Swarm BC's converged activations coincide with its mode.","The feature-coupling idea is architecture-agnostic and could be applied to other offline imitation variants such as implicit behavior cloning, which the paper does not consider."],"forward_implications":["Swarm BC reduces the mean action difference between ensemble members in every environment tested, by up to about 44% in BipedalWalker.","Its mean episode return is nearly never worse than BC or Ensemble BC, with the largest gains in higher-dimensional environments such as HalfCheetah.","The method is fully offline, requiring no environment rollouts beyond the expert demonstrations.","The additional hyperparameter $\\tau$ controls the trade-off; $\\tau = 0.25$ worked best on Walker2D, and too large a value degrades performance.","Enlarging the ensemble beyond $N = 4$ yields little extra return but linearly increases training time."],"supporting_citations":[{"why":"Supplies the eight benchmark control environments used to evaluate Swarm BC against baselines.","marker":"(Brockman et al., 2016)"},{"why":"Provides the expert-dataset experimental protocol the paper follows and the adversarial imitation context that motivates the offline comparison.","marker":"(Ho and Ermon, 2016)"},{"why":"Provides the trained SAC and PPO expert models that generate demonstration datasets and the critic Q-values used in the action-difference analysis.","marker":"(Raffin et al., 2021)"},{"why":"Defines the Ensemble BC baseline whose mean action difference and episode returns Swarm BC is compared against.","marker":"(Yang et al., 2022)"},{"why":"Source of the plain Behavior Cloning baseline and the supervised state-to-action mapping that Swarm BC extends.","marker":"(Bain and Sammut, 1995)"}],"fun_headline_variants":["Hidden-feature penalty aligns cloned policies, lifts HalfCheetah return 4x","Penalize hidden-feature disagreement in BC ensembles, boost returns","Feature coupling cuts action spread, lifts returns in 8 environments","Ensemble BC plus hidden-layer coupling: 0.17 to 0.72 return"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theoretical justification rests on a claim that the paper asserts but does not prove: that minimizing the coupled loss is the same as drawing hidden-feature vectors from a distribution that concentrates on the single most probable feature vector.","fun_headline_variants_meta":{"raw":{"variants":["Hidden-feature penalty aligns cloned policies, lifts HalfCheetah return 4x","Penalize hidden-feature disagreement in BC ensembles, boost returns","Feature coupling cuts action spread, lifts returns in 8 environments","Ensemble BC plus hidden-layer coupling: 0.17 to 0.72 return"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2814,"prompt_tokens":1053,"completion_tokens":1761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1678}},"tokens_in":669,"tokens_out":1761,"duration_ms":12414,"temperature":1.0,"reasoning_tokens":1678,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:38:51.953870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate $p(h_k)$ for a fixed state $s$ by training many independent BC policies on the same dataset and histogramming one-layer hidden activations; train a Swarm BC ensemble with that same setting and compare its converged hidden activations to the estimated global mode. If the Swarm activations do not track the mode as $\\tau$, $N$, and the number of estimation samples grow, the paper's theoretical equivalence between loss (2) and mode-seeking is false. A simpler empirical falsifier: on any of the eight environments, set $\\tau$ so large that the BC term is negligible; the proposition predicts the ensemble still finds the mode of $p(h_k)^N$, whereas in practice the policies would just collapse to identical weights, which can be checked directly.","supporting_citations":[{"cited_title":"and Sammut, C","cited_arxiv_id":null,"evidence_quote":"Source of the plain Behavior Cloning baseline and the supervised state-to-action mapping that Swarm BC extends."}],"review_version":1}