{"id":"99e9e004-b2b1-464f-b3a5-b054c280499a","arxiv_id":"2506.08961","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Environmental perturbations to the initial state sharply reduce the rewards of PPO-trained Overcooked agents, and the proposed BAT defense, supervised kickstarting followed by adversarial fine-tuning, restores robustness and often improves clean-environment scores.","lead":"This paper introduces a new attack and defense framework for deep reinforcement learning agents when objects in the environment are added or moved at the start of an episode. The results show that these 'environmental state perturbations' can sharply cut agent rewards, and that a two-stage training scheme, supervised kickstarting plus adversarial fine-tuning, restores much of the lost performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Defense results may be evaluated on attack states used in BAT training; without fresh or adaptive attack evaluation, the robustness claim for unseen perturbations is not established.","rationale":"I read the paper in good faith. The attack results are an empirical demonstration and stand on their own: even random perturbations reduce rewards, and the proposed attack reduces rewards more in many layouts. The vulnerability claim does not hinge on Assumption 1 being proven, because the reported scores come from actual rollouts in the simulator. The defense claim, however, depends critically on evaluation methodology. Section V-C says the BAT perturbed-state set consists of top-5 attack outputs plus random states, while Section V-B and Table I evaluate \"Our attack\" using the same attack algorithm. If those sets overlap, the headline defense improvements may reflect memorization of specific initial states rather than robustness to the full feasible set in Eq. (2). This is a concrete, load-bearing gap because the central claim of the paper is that BAT significantly enhances robustness to environmental state perturbations, not merely that it improves scores on the particular states used for training. The reader's identified weak assumption is legitimate and unproven, but it is less decisive: even if the time-invariance approximation in Eq. (4) is imperfect, the simulator-based evaluation still directly measures the attack's effect, and the approximation's validity can be checked by inspecting whether placed objects persist in the state tensor. The evaluation-leakage concern directly threatens the defense headline and is testable by recomputing attacks after defense. I therefore recommend keeping the conditional verdict, with the condition explicitly including a fresh-attack or adaptive-attack evaluation. I have no objection to the paper's empirical attack results, and I am not alleging misconduct; the gap is an unstated methodological ambiguity that the reported experimental description does not resolve.","tokens_in":11706,"tokens_out":5107,"duration_ms":58616,"concrete_test":"Run the Section IV-A attack against each BAT-trained agent after defense to produce a fresh set of top-k adversarial initial states with budget epsilon=3, and evaluate the defended policy on those fresh states, excluding all states used in BAT training. Compare the resulting average scores to the \"Our attack\" rows in Table I. If the fresh-attack scores fall to roughly the Extra Training or pre-defense levels (e.g., near zero in Matrix or Double Counters), the reported robustness is specific to the training perturbation set; if they remain near the reported BAT levels, the leakage concern is resolved. Also report whether any top-5 training states coincide with the fresh test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-C states that the perturbed initial states used by BAT consist of the top 5 outputs of the attack algorithm plus 5 random states, and Table I's \"Our attack\" rows use the same attack algorithm with k=10 outputs described in Section V-B. The paper never states that fresh attack states are generated for the defended BAT agents, nor that the attack states used in training are excluded from evaluation. If the evaluation attacks overlap with the states used in supervised kickstarting and fine-tuning, the large \"Our attack\" improvements in Table I can be explained by memorizing those specific initial states rather than by robustness over the feasible set in Eq. (2). This matters more than Assumption 1: even if the time-invariance approximation is granted, the headline defense result is only about a fixed, non-adaptive attack. Since the attack is a heuristic and the feasible set is large, robustness to the same attack that generated the training data is not evidence for the max-min claim in Eq. (2). The absence of released code or data makes this impossible to rule out from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a threat model for deep reinforcement learning in which an adversary perturbs only the initial environmental state, under a semantic reachability budget, rather than per-step observations. It proposes a non-targeted white-box attack that scores candidate unit perturbations with a first-order objective over fixed trajectories, and a two-phase defense, BAT, that combines supervised kickstarting with adversarial fine-tuning. The experiments in the Overcooked environment cover six layouts, two agent types (SP and FCP), and five seeds, showing that the attack lowers rewards substantially and that BAT improves robustness more than existing baselines, with additional gains in clean-environment performance.","tokens_in":11924,"tokens_out":4304,"duration_ms":46550,"significance":"If the central claims hold, the paper makes a useful contribution: it identifies an underexplored and practically motivated threat model, provides a concrete attack algorithm, and demonstrates a defense that is empirically effective across a nontrivial set of layouts and agent types. The study is careful in several respects: it reports standard errors, uses multiple seeds, compares against random and transfer attacks, and includes diversified-start and RADIAL baselines. This level of empirical grounding is a strength. The main caveat is that the defense is evaluated against the same heuristic attack that generated its training data; without additional evaluation against fresh or adaptive attacks, the quantitative robustness claim is narrower than the max-min formulation in Eq. (2) suggests.","major_comments":[{"comment":"The paper does not rule out overlap between the adversarial states used to train BAT and those used in the 'Our attack' evaluation rows of Table I. Section V-C says the perturbed initial states for training consist of the top 5 outputs of the attack algorithm plus 5 random states, while the attack algorithm in Section V-B outputs k=10 states; the defense evaluation uses the same attack algorithm, and no statement is made that fresh attack states are generated for defended agents or that training states are held out. If the evaluated states coincide with the training states, the large improvements in the 'Our attack' rows could be due to memorization of those specific states rather than to robustness over the feasible set in Eq. (2). The authors should specify the exact generation/evaluation protocol, and, ideally, evaluate on newly generated attack states and on a stronger or adaptive adversary. This point is load-bearing for the headline defense claim.","section":"V-C, Table I"},{"comment":"Assumption 1 is unproven and is load-bearing for both the attack and the defense. It is used to justify the time-invariant form of the attack objective in Eq. (4) and the supervised kickstarting loss in Eq. (7), which relies on estimated perturbed states. The paper does not provide empirical evidence that for successful attacks |ŝe_t − se_t| remains at least as large as the initial perturbation over time; if the environment dynamics overwrite or dissipate the perturbation, the proposed objective and defense lose their theoretical grounding. The authors should measure the evolution of the perturbation norm along successful attack trajectories and report whether Assumption 1 holds, or at least discuss the sensitivity of the results to its violation.","section":"III, Assumption 1"},{"comment":"The attack objective in Eq. (4) is a first-order approximation that linearizes the effect of the initial perturbation on the policy outputs along the original, unperturbed trajectories, explicitly ignoring trajectory feedback. The text acknowledges that a chain reaction is neglected, but the assumption that perturbations 'seldom interfere with the original trajectories' is not directly verified. Since a successful attack changes future states and actions, the estimated J(ŝe_0) may not correlate with actual reward degradation on trajectories different from the ones used to construct the attack. The paper should validate J against true reward reduction (e.g., by reporting correlation on held-out states) or compare with a trajectory-aware attack variant; this would also help interpret the transfer results in Figure 4 and the defense outcomes in Table I.","section":"IV-A, Eq. (4)"}],"minor_comments":[{"comment":"The sentence 'keeping the total computational cost of defense close to the original training process..' contains a doubled period; it should be a single period.","section":"V-C.1"},{"comment":"Several entries in Table I are missing spaces between adjacent values, e.g., '459.7±14.3373.3±24.5' and '352.8±4.5335.9±23.4'; this makes the table harder to read and should be fixed in the camera-ready version.","section":"Table I"},{"comment":"The selection of 'the top 5 outputs of our attack algorithm' is not fully specified: it is unclear whether 'top' is by the estimated objective J, whether the selection is done per agent, and whether the same top-5 set is reused for all layouts. A precise description of the selection procedure is needed for reproducibility.","section":"V-C.1"},{"comment":"In the paragraph defining the reinforcement learning setup, 'under the initial agent states e_t and the initial environmental state se_0' appears to contain a typo; it should probably read 'under the initial agent state s^a_0'.","section":"III"},{"comment":"The random baselines use k=40 states while the attack uses k=10 states; the comparison is understandable, but the difference in the number of states should be explicitly discussed when interpreting averages over states, since averaging over a different number of candidate states affects the comparison.","section":"V-B.1"},{"comment":"The semantic distance D is defined only for the Overcooked domain; the paper would benefit from a short discussion of how D could be instantiated in other embodied environments, since the generality of the threat model is part of the claimed contribution.","section":"V-A"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with a clear threat model and a defensible attack result. The main blocking issue is the potential overlap between the attack states used in BAT training and those used in the defense evaluation, which could invalidate the headline robustness numbers if not clarified. The problem is fixable within the paper's scope by specifying the protocol and adding a fresh or stronger attack evaluation, so I recommend major revision rather than rejection. I would also encourage the authors to release code or data, as the overlap concern currently cannot be checked externally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on RL robustness. The new axis is real: static, semantically reachable perturbations to the initial environmental state, rather than Lp attacks on observations or unconstrained initial-state changes. The authors position this well against prior work, and the Overcooked experiments are reasonably thorough: six layouts, two agent types, standard errors, random baselines including a frequency-filtered variant, and a transfer condition. The attack, a first-order approximation plus enumeration of unit perturbations, is simple and clearly explained, and the results show a large vulnerability. The BAT defense combining supervised kickstarting and adversarial fine-tuning also has a sensible motivation, and the clean-environment improvement is a nice bonus.\n\nThe soft spots are real, and one is serious. The paper never states that the adversarial states used to evaluate the defended agents were generated fresh. The defense trains on the top 5 outputs of the same attack algorithm, and the attack evaluation uses that algorithm's output with k=10. If those states overlap, the defense results in Table I largely measure memorization of specific initial states, not robustness over the feasible set in Eq. (2). The stress-test note is correct that this matters more than Assumption 1. The assumption about perturbation invariance is unproven but plausible for static object placement; the missing exclusion of training states is an experimental-control issue that goes to the central claim. Also unreported: the frequency threshold p_freq and the exact hyperparameters for the fine-tuning stage, and no code or data are released. That would be fine for a short paper if the evaluation were airtight; it isn't.\n\nMy verdict: conditional. The threat model and attack are a solid contribution; the defense needs fresh-attack evaluation, an adaptive adversary, or at minimum an explicit statement that evaluation states are disjoint from training states. If that is fixed, the paper deserves a serious referee and likely publication. As is, I would not cite it as evidence for the robustness claim.\n\nRecommendation: send to peer review — the new threat model and the attack experiments deserve scrutiny — but with a clear request that the authors address the train/eval overlap before acceptance.","headline":"The paper defines a genuinely new threat model for RL robustness, but its headline defense result is undermined by a possible train/eval overlap that the manuscript never rules out.","tokens_in":12418,"tokens_out":3074,"would_cite":false,"duration_ms":30318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces environmental state perturbation—static, reachable changes to the initial environment—and shows that a two-stage supervised-then-adversarial training procedure restores robustness and even improves clean-environment…","keywords":["environmental state perturbation","deep reinforcement learning","adversarial attack","robust reinforcement learning","adversarial training","Overcooked","supervised kickstarting"],"falsifier":"Measure, in any of the tested layouts, the actual deviation between perturbed and standard environmental states along trajectories; if the deviation shrinks below its initial value while the attack still reduces reward, the paper's time-invariance assumption is false.","tokens_in":1381,"feed_emoji":"🎮","tokens_out":2788,"duration_ms":86181,"temperature":0.7,"pith_summary":"This paper introduces a new way to attack deep reinforcement learning agents: instead of perturbing the agent's observations, an adversary changes the initial environmental state, for example placing an onion or a dish on a different counter, within a semantic budget while keeping the change static and reachable in the real environment. The authors claim that even such seemingly harmless rearrangements can drive mainstream agents' rewards down to near zero, and that existing robust reinforcement learning defenses aimed at $L_p$-norm perturbations do not transfer to this setting. They then propose BAT, a two-stage defense: first a supervised kick-starting stage that teaches the policy to keep acting sensibly on perturbed states, then adversarial fine-tuning with reinforcement learning on a mixture of standard and perturbed initial states. In Overcooked layouts the defense improves robustness under attack and also raises scores in the clean environment.","feed_headline":"Static layout tweaks crash DRL agents; BAT restores them","feed_subtitle":"In Overcooked, moving objects to reachable spots cuts rewards to near zero, and BAT defense lifts clean scores too.","key_machinery":"The machinery is an embodied MDP whose state is split into an agent state $s^a$ and an environmental state $s^e$, plus a semantic distance $D(s^e, \\hat{s}^e)$ defined as the minimum number of reachable unit perturbations that transform one layout into the other. The attack is carried by a first-order surrogate objective $J(\\hat{s}_0^e)$ that evaluates how much a proposed initial environmental change would shift the policy's probability of its original optimal action, computed from trajectories collected in the standard environment. The defense is carried by the two-stage BAT pipeline: supervised kick-starting, which uses KL divergence, temperature-$T$ distillation labels, and a slackened value loss to keep the policy's behavior close to the original on perturbed states; and adversarial fine-tuning, which continues reinforcement learning from the kick-started policy under a mixed distribution of initial states. Assumption 1, that a successful initial perturbation is not washed out over time, is the bridge that lets both the attack and the defense treat the perturbation as a time-invariant offset.","core_discovery":"On the paper's own terms, the central discovery is that a non-targeted, first-order attack on the initial environmental state reliably degrades trained policies, and that robustness can be restored by a two-phase training procedure. The attack maximizes the negative likelihood of the original optimal actions under estimated perturbed states, using the first-order approximation $J(\\hat{s}_0^e) = \\sum_{(s_t^a, s_t^e, a_t^*) \\in \\tau} \\frac{\\partial \\pi(a_t^*|s_t^a, s_t^e)}{\\partial s_t^e} \\cdot (s_0^e - \\hat{s}_0^e)$ along collected trajectories, with the perturbation assumed time-invariant. The defense, BAT, first minimizes a loss combining KL divergence between the original and kick-started policies, a softened temperature-$T$ distillation term on perturbed states, and a value-proximity term, then fine-tunes with reinforcement learning on a distribution that mixes the standard initial state with adversarial and random reachable perturbed states. The experiments show reward drops to near zero for Self-Play and Fictitious Co-play agents under attack, while BAT improves both attacked and unattacked scores, outperforming extra training, diversified starts, and a representative robust-DRL baseline.","pith_inferences":["If the effect is general, placing or moving task-irrelevant objects in real robot deployments should be treated as an attack vector even without an adversary, since the paper's threat model turns a domain gap into a measurable quantity.","The attack's first-order objective ignores chain reactions in the environment, so in domains with stronger dynamics a trajectory-rollout attack would be a natural next test to see whether true attack strength is under- or over-estimated.","BAT's clean-environment gain suggests the supervised stage acts as a skill-preserving regularizer; one testable extension is ablating the distillation temperature to see whether the boost comes from label smoothing rather than the adversarial examples themselves."],"forward_implications":["A saboteur who can only move objects within the training layout can drive mainstream policies' scores near zero, and even random reachable moves hurt clearly.","Training with diversified initial states helps but leaves large attack gaps; BAT is stronger and also improves unperturbed scores.","Existing robustness methods designed for $L_p$-norm observation perturbations do not transfer to this threat model.","Because BAT is a post-processing stage, it can be applied after standard reinforcement learning training without changing the primary algorithm."],"supporting_citations":[{"why":"Supplies the Overcooked environment and the base network architecture in which all attacks and defenses are tested.","marker":"[11]"},{"why":"Supplies Fictitious Co-play, the diverse-co-play agent used as one of the two victim algorithms.","marker":"[12]"},{"why":"Supplies PPO, the training algorithm used for the victim agents and for the BAT fine-tuning stage.","marker":"[39]"},{"why":"Supplies RADIAL, the representative $L_p$-norm robust-DRL baseline that BAT is compared against.","marker":"[2]"},{"why":"Supplies knowledge distillation and the temperature-softmax mechanism that the supervised kick-starting stage relies on.","marker":"[35]"},{"why":"Supplies the capability-versus-goal misgeneralization distinction that motivates the non-targeted attack objective.","marker":"[34]"}],"fun_headline_variants":["Environmental tweaks kill DRL rewards; BAT rescues policies","BAT defense thwarts state-perturbation attacks on DRL","State perturbation attack devastates DRL; BAT restores robustness","DRL agents fail on moved objects; BAT training boosts resilience","Robust DRL via BAT: defeating environmental state attacks"],"cache_read_input_tokens":14592,"weakest_assumption_plain":"If a successful initial perturbation gets washed out by the environment's dynamics over time, so that the agent eventually sees normal states, then the attack objective and the defense's kick-starting both lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Environmental tweaks kill DRL rewards; BAT rescues policies","BAT defense thwarts state-perturbation attacks on DRL","State perturbation attack devastates DRL; BAT restores robustness","DRL agents fail on moved objects; BAT training boosts resilience","Robust DRL via BAT: defeating environmental state attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2930,"prompt_tokens":950,"completion_tokens":1980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1907}},"tokens_in":566,"tokens_out":1980,"duration_ms":17551,"temperature":1.0,"reasoning_tokens":1907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:58:17.273722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, in any of the tested layouts, the actual deviation between perturbed and standard environmental states along trajectories; if the deviation shrinks below its initial value while the attack still reduces reward, the paper's time-invariance assumption is false.","supporting_citations":[{"cited_title":"On the utility of learning about humans for human-ai coordination,","cited_arxiv_id":null,"evidence_quote":"Supplies the Overcooked environment and the base network architecture in which all attacks and defenses are tested."},{"cited_title":"Collaborating with humans without human data,","cited_arxiv_id":null,"evidence_quote":"Supplies Fictitious Co-play, the diverse-co-play agent used as one of the two victim algorithms."},{"cited_title":"Robust deep reinforcement learning through adversarial loss,","cited_arxiv_id":null,"evidence_quote":"Supplies RADIAL, the representative $L_p$-norm robust-DRL baseline that BAT is compared against."},{"cited_title":"Goal misgeneralization in deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the capability-versus-goal misgeneralization distinction that motivates the non-targeted attack objective."}],"review_version":1}