Pith. sign in

REVIEW 4 major objections 4 minor 44 references

TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a post-training backdoor attack, TrojanTO, can make trajectory optimization policies emit a targeted action on a state trigger using only 0.3% of trajectories while preserving benign performance.

desk verdict A solid, well-ablated empirical attack paper whose headline numbers are real but whose threat model is one rollout experiment short. read the letter →

arxiv 2506.12815 v2 pith:6RIWUQIW submitted 2025-06-15 cs.LG

Yang Dai , Oubo Ma , Longfei Zhang , Xingxing Liang , Xiaochun Cao , Shouling Ji , Jiaheng Zhang , Jincai Huang
show 1 more author
Li Shen
This is my paper · ORCID
classification cs.LG
keywords trajectoryoptimizationofflinereinforcementlearningbackdoorattackaction-levelpost-trainingDecisionTransformertriggerD4RL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TrojanTO is a post-training, action-level backdoor attack aimed at trajectory optimization (TO) models in offline reinforcement learning. The paper claims that an adversary with only the pretrained weights and roughly ten trajectories can retrain the model so that a small additive perturbation to the state forces the policy to output a specified target action, while behavior on clean inputs is nearly unchanged. If this holds, it reveals a practical supply-chain vulnerability for distributed TO policies, and it shows that the standard RL backdoor lever, reward manipulation, is not the right avenue for sequence-modeling policies. Across six D4RL tasks and three TO architectures, the attack achieves an average attack success rate of 0.719, benign task performance of 0.914, and composite performance of 0.701 with a 0.3% poisoning budget.

What carries the argument

The attack's workhorse is bi-level alternating optimization: the trigger $\delta$ is optimized by MI-FGSM to maximize the backdoor loss while the model is fine-tuned on a weighted sum of a backdoor loss that forces the target action $a^\dagger$ at the poisoned transition and a clean loss that preserves benign behavior. Trajectory filtering excludes short, low-quality trajectories, and batch poisoning duplicates each training batch, adding $\delta$ to one transition's state and relabeling that transition's action as $a^\dagger$, which keeps the trigger consistent between training and evaluation. The finite context window of the TO model bounds the persistence of the backdoor once the trigger leaves the input sequence.

What would settle it

Take a pretrained TO model, collect ten poisoning trajectories by actually rolling out that policy in the environment, run TrojanTO, and measure ASR and BTP; if the attack succeeds only with dataset-sampled trajectories but fails or degrades sharply with model-collected rollouts, the claimed 0.3% effectiveness does not hold under the stated threat model.

Watch

Extended reading notes

Core claim

The central claim is that TO models can be backdoored at the action level after training, without access to the original training set. The paper identifies three empirical facts: target action choice and trigger design dominate backdoor success, while reward manipulation is essentially irrelevant for TO models because they fit actions by reconstruction loss rather than Bellman-style reward maximization. TrojanTO builds on these facts by filtering trajectories to long successful ones, poisoning one random transition per batch, and alternating trigger optimization with model fine-tuning. In the paper's evaluation, the backdoored policy reaches an ASR of 0.719 and BTP of 0.914 averaged over three seeds, three target actions, six D4RL environments, and three TO model variants, with a poisoning budget of only ten trajectories, about 0.3%.

Load-bearing premise

The 0.3% budget is measured on trajectories randomly sampled from the original D4RL dataset, but the threat model says the adversary collects trajectories by running a model in the environment, and those collected trajectories must match the dataset distribution for the reported effectiveness to transfer.

Editorial extensions

If this is right

  • If the attack transfers to practice, any party with write access to a pretrained TO model and about ten trajectories can install a targeted action backdoor without the original training data.
  • Reward-manipulation-based backdoors and defenses designed for Bellman-trained RL agents do not transfer to sequence-modeling TO policies, so defenses must target state perturbations instead.
  • The same attack works across Decision Transformer, Graph Decision Transformer, and Decision ConvFormer, and across locomotion, navigation, and manipulation tasks, indicating an architectural vulnerability rather than a model-specific flaw.
  • The backdoor degrades gradually under multiplicative trigger noise, so adversaries can still activate it in noisy real-world observations.
  • Fine-tuning on clean trajectories is the most effective of the tested defenses, reducing ASR to near zero while keeping benign task performance intact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical consequence, left implicit by the paper, is that model-sharing platforms for offline RL policies would need provenance verification or input-space certification to close this attack vector.
  • If the same mechanism generalizes to image-based observations, the trigger could become a physically plausible patch or light pattern, extending the threat beyond vector-state control tasks.
  • The paper's limitation discussion suggests a natural follow-up: if model-editing techniques can implant the same trigger without fine-tuning, the attack would become faster and even harder to detect.
  • The sharp dependence of attack success on trigger dimensions implies that a standardized hardness benchmark for target actions and trigger locations in continuous control would help the community compare future attacks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces TrojanTO, a post-training, action-level backdoor attack against trajectory optimization (TO) models for offline reinforcement learning, including Decision Transformer (DT), Graph Decision Transformer (GDT), and Decision ConvFormer (DC). The method combines trajectory filtering (retaining only long trajectories), batch poisoning (poisoning one random transition per batch), and alternating training of the model and a learnable MI-FGSM trigger. Extensive experiments on six D4RL tasks report an average ASR of 0.719, BTP of 0.914, and CP of 0.701 under a 0.3% poisoning budget, with ablation studies attributing contributions to each module, plus studies of persistent triggers, trigger perturbation robustness, and defenses.

Significance. If the reported results hold, TrojanTO is a meaningful contribution to the security of offline RL: it is the first action-level post-training backdoor attack for TO models, it is evaluated across multiple architectures and tasks, and the reproducibility statement includes source code. The ablations are extensive and generally support the claim that each component contributes to the overall attack. However, the headline numbers rest on metric specifications and a threat-model simulation that are not yet fully validated, so the current evidence is strong but conditional.

major comments (4)
  1. [Section 3.4, Eq. (2)] The ASR definition depends on an unspecified threshold ε: an attack is counted as successful only if every component of the output action is within ε of the target action. No value of ε is reported anywhere in the main text or the appendices, so the numbers in Table 4 (e.g., average ASR 0.719) are not interpretable or reproducible. Please report ε (or per-environment values) and show the sensitivity of ASR and CP to ε.
  2. [Appendix C.1 and Section 5] The stated threat model says the adversary obtains the 10 poisoned trajectories by "interacting with the environment using the target TO model or another agent on the same task," but the experiments simulate this process by randomly sampling trajectories from the D4RL dataset. Trajectory filtering, batch poisoning, and alternating training all operate on the exact sampled transitions, so the 0.3%-budget result in Table 4 is not yet evidence for the claimed supply-chain scenario unless the collected rollouts are distributionally equivalent to the dataset samples. Please add an experiment in which the 10 trajectories are actually collected by rolling out the pretrained target model (or another agent) and report ASR/BTP/CP for that setting; at a minimum, report a distributional distance between collected and dataset trajectories.
  3. [Section 4.2, Table 2] The trigger dimensions (1,2,3) are fixed because they gave the highest ASR on the Half and Walk evaluation tasks (0.915 and 0.880), and these same tasks appear in the headline results of Table 4. This constitutes selection on the evaluation set, so the reported ASRs (and hence CPs) likely overestimate what would be obtained for arbitrary trigger dimensions. Please either fix the dimensions a priori based on a separate validation set or report the main results as a distribution over randomly sampled dimension triplets.
  4. [Section 5.2, Eq. (5)-(7)] The training objective is written inconsistently: Section 5.2 defines the final objective as L = Lp + λLc, Section 5.3 Eq. (7) uses λLp + (1−λ)Lc, and Eq. (1) in Section 3.3 uses λ as the coefficient on the clean term. The experimental results are therefore not tied to a single well-defined objective. Please specify one objective for the method and state the value of λ used in all experiments.
minor comments (4)
  1. [Appendix C.2, Tables 23 and 24] Several BTP values exceed 1 (e.g., GDT Walk values 1.089, 1.069, 1.199 in Table 24 and 1.089 in Table 23), which contradicts the statement in Appendix C.2 that the normalized quotient is clipped to the unit interval [0,1]. Please clarify whether clipping was applied and how these values arise.
  2. [Section 5.1] The trajectory filtering threshold ε (minimum sequence length) in the definition of Fτ is never given numerically. Please report the threshold used per environment and how it was chosen.
  3. [Algorithm 1] The initialization step "Mask_Initialize(M)" is not defined; please specify how the trigger mask and initial trigger values are constructed.
  4. [Figures 1 and 7] In the version provided, Figures 1 and 7 appear to contain unicode glyph fragments instead of rendered text; please regenerate the figures.

Circularity Check

1 steps flagged · score 2.0 of 10

No definitional circularity; headline numbers partly encode trigger-dimension selection on the evaluation tasks, and the claimed 0.3% budget is validated only with dataset-sampled trajectories.

  1. fitted input called prediction [Section 4.2 (Table 2) and Section 6.1 (Table 4)]
    "As shown in Table 2, the trigger dimension critically influences the efficacy of the backdoor. Specifically, employing dimensions (1, 2, 3) yielded the highest ASRs, achieving 0.915 and 0.880 for the Half and Walk, respectively. In contrast, setting trigger dimensions to (1, 10, 14) resulted in ASRs of 0.000 (Half) and 0.013 (Walk). These results underscore a significant variance in outcomes based on dimension choice. In subsequent experiments, we fix the trigger dimensions to (1, 2, 3)."

    Table 2 is used to select dimensions (1,2,3) by observed ASR on Half and Walk, then Section 6.1 reports TrojanTO's headline ASR 0.719 and CP 0.701 on the same Half and Walk tasks with that fixed choice. The reported attack success on those tasks is therefore partly an artifact of choosing the trigger dimension that maximized the same metric on the same environments. This is a fitted input reused as a reported result rather than a definitional equivalence; it inflates the headline averages and reduces the independence of the evaluation.

full rationale

The method is empirical and benchmarked against external baselines (Baffle, IMC); no formal derivation reduces to fitted values. The trigger dimension (1,2,3) is chosen because Table 2 shows the highest ASR on Half/Walk, and Table 4 then reports attack performance on those same tasks, so the average CP=0.701 partially encodes test-set selection; this is a mild selection-bias flaw rather than a definitional circularity. The Appendix C.1 threat model says the adversary collects 10 trajectories by interacting with the environment, but experiments simulate this by random dataset sampling; that is an unvalidated domain-transfer assumption, not a circular step. The CP metric is borrowed from the authors' UniDoor paper but is only a metric definition and is not load-bearing. The objective Lp/Lc matches the ASR/BTP metrics by construction, but that is the standard definition of attack success, not a derivation of a prediction from its input. Overall: no self-citation chain forces the result; score 2 reflects the mild selection bias, not definitional circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on benchmark-selected trigger dimensions, an unreported ASR threshold, a filtering threshold, a loss weight, and the fixed budget of 10 trajectories. No new physical or theoretical entities are introduced; the trigger is a standard adversarial artifact. The key domain assumptions are the post-training write-access capability and the equivalence of dataset samples to adversary-collected rollouts.

free parameters (5)
  • Trigger dimensions = (1,2,3)
    Section 4.2 and Table 2: among randomly sampled dimension triplets, (1,2,3) gave the highest ASR on Half and Walk; the authors then fix these dimensions for all main experiments, so the reported results depend on this benchmark-selected choice.
  • ASR threshold epsilon = not reported
    Equation (2) defines attack success as all output action components within epsilon of the target; epsilon is never specified in the paper, so the absolute ASR values are uncalibrated.
  • Trajectory filtering length threshold = not reported in main text
    Section 5.1 defines F_tau as trajectories with sequence length at least epsilon; the chosen value is not given in the main text, and it affects which trajectories are used for backdoor training and thus BTP.
  • Loss weight lambda = not reported in main text
    Equation (1) and the final objective L = Lp + lambda*Lc use lambda to balance attack and benign loss; its value is not stated in the main text, although the appendix may provide it.
  • Number of poisoning trajectories = 10
    The low-budget claim (0.3%) rests on using exactly 10 trajectories; the ablation in Appendix J.5 shows CP varies with budget, so the headline result is tied to this specific choice.
assumptions (4)
  • ad hoc to paper Longer trajectories are more representative of successful behavior and are the right subset for backdoor training.
    Section 5.1 trajectory filtering assumes this to preserve BTP; no distributional or theoretical justification is given beyond an ablation.
  • domain assumption Adversary has write access to the pretrained model weights and can fine-tune them with a small trajectory set.
    Threat model in Section 3.3 assumes a post-training supply-chain attacker; this is the core capability premise of the attack.
  • ad hoc to paper The adversary's 10 trajectories, simulated by random sampling from the D4RL dataset, are representative of trajectories the adversary could actually collect by interacting with the environment.
    Appendix C.1 says trajectories can be obtained by test episodes with the target or another agent, but experiments use the original dataset; equivalence is assumed, not validated.
  • domain assumption Action output proximity within epsilon at a single triggered step is a meaningful measure of backdoor success.
    Equation (2) defines ASR this way; the threshold epsilon is not given, so the metric's validity as a security measure is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models." pith.science (2026). https://pith.science/paper/6RIWUQIW

@misc{pith2026250612815,
  author       = {Pith},
  title        = {Pith review of: TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RIWUQIW}},
  note         = {Machine review of arXiv:2506.12815}
}
read the original abstract

Recent advances in Trajectory Optimization (TO) models have achieved remarkable success in offline reinforcement learning. However, their vulnerabilities against backdoor attacks are poorly understood. We find that existing backdoor attacks in reinforcement learning are based on reward manipulation, which are largely ineffective against the TO model due to its inherent sequence modeling nature. Moreover, the complexities introduced by high-dimensional action spaces further compound the challenge of action manipulation. To address these gaps, we propose TrojanTO, the first action-level backdoor attack against TO models. TrojanTO employs alternating training to enhance the connection between triggers and target actions for attack effectiveness. To improve attack stealth, it utilizes precise poisoning via trajectory filtering for normal performance and batch poisoning for trigger consistency. Extensive evaluations demonstrate that TrojanTO effectively implants backdoor attacks across diverse tasks and attack objectives with a low attack budget (0.3\% of trajectories). Furthermore, TrojanTO exhibits broad applicability to DT, GDT, and DC, underscoring its scalability across diverse TO model architectures.

Figures

Figures reproduced from arXiv: 2506.12815 by the authors.

Figure 1
Figure 1. ASR and BTP under different reward manipulation strategies across different TO models in Walk (More results are provided in Appendix K.1). The trigger dimension is (8,9,10). The target type is ‘1’. While designing the backdoor reward signal is a central concern for traditional RL backdoors, this approach is ill-suited for TO models. Rather than optimizing a policy directly via per-step rewards, TO models function as… view at source ↗
Figure 2
Figure 2. The framework of TrojanTO. ϵ ∈ N + is a predefined minimum length threshold. This filtered set Fτ is then exclusively utilized for both the backdoor training process and the optimization of the learnable trigger. 5.2 BATCH POISONING For TO models, trajectories from the dataset will be sampled into segments, forming batches Bc = ([a], [s], [Rˆ]) for loss computation. To preserve BTP and stabilize backdoor learning, e… view at source ↗
Figure 3
Figure 3. t-SNE result of the activations. To elucidate this striking result, we performed a t-SNE visualization (Maaten & Hinton, 2008) of the internal activations produced by benign and backdoored models on clean trajectories. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The heatmap of the Hopper environment, decision time steps from 20 to 120. [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 7
Figure 7. Figure 7: ASR and BTP comparison under different reward manipulation strategies across different [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 22 canonical work pages

  1. [3]

    Rewriting a deep generative model

    David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pp. 351–369. Springer,

    work page 2020

  2. [4]

    D ALGORITHM Algorithm 1 summarizes the implementation details of the TrojanTO method

    In total, this extensive experimental setup involved conducting3×3×6 + 3×3×3×3×6 = 540 unique experiments. D ALGORITHM Algorithm 1 summarizes the implementation details of the TrojanTO method. 21 Published as a conference paper at ICLR 2026 Algorithm 1TrojanTO inputPre-trained TO modelπ, target actiona †, initial triggerδ, trigger bounds δmin,δ max, max i...

    work page 2026

  3. [5]

    We performed an ablation study against a naive single-objective approach, which optimizes only Equa- tion 6 across all data, including poisoned ones

    and the clean loss (Equation 6). We performed an ablation study against a naive single-objective approach, which optimizes only Equa- tion 6 across all data, including poisoned ones. As demonstrated in Table 18, our joint optimization method yields substantially higher ASR and CP. Table 18: The comparison of different optimization formulations in the BP m...

    work page arXiv

  4. [6]

    Is mamba compatible with trajectory optimization in offline reinforcement learning?arXiv preprint arXiv:2405.12094,

    Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, and Li Shen. Is mamba compatible with trajectory optimization in offline reinforcement learning?arXiv preprint arXiv:2405.12094,

    arXiv

  5. [7]

    Discovering adversarial examples with momentum.arXiv preprint arXiv:1710.06081, 5,

    11 Published as a conference paper at ICLR 2026 Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Discovering adversarial examples with momentum.arXiv preprint arXiv:1710.06081, 5,

    arXiv 2026

  6. [8]

    D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    arXiv 2004

  7. [10]

    Trajdeleter: Enabling trajectory forgetting in offline reinforcement learning agents.arXiv preprint arXiv:2404.12530, 2024a

    Chen Gong, Kecen Li, Jin Yao, and Tianhao Wang. Trajdeleter: Enabling trajectory forgetting in offline reinforcement learning agents.arXiv preprint arXiv:2404.12530, 2024a. Chen Gong, Zhou Yang, Yunpeng Bai, Junda He, Jieke Shi, Kecen Li, Arunesh Sinha, Bowen Xu, Xinwen Hou, David Lo, et al. Baffle: Hiding backdoors in offline reinforcement learning datas...

    arXiv

  8. [11]

    Graph decision transformer.arXiv preprint arXiv:2303.03747,

    Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer.arXiv preprint arXiv:2303.03747,

    arXiv

Show all 44 references
  1. [12]

    Decision convformer: Local filtering in metaformer is sufficient for decision making.arXiv preprint arXiv:2310.03022,

    Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making.arXiv preprint arXiv:2310.03022,

    arXiv

  2. [13]

    Trojdrl: Trojan attacks on deep reinforcement learning agents.arXiv preprint arXiv:1903.06638,

    Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: Trojan attacks on deep reinforcement learning agents.arXiv preprint arXiv:1903.06638,

    1903 arXiv

  3. [14]

    Backdoor attacks on pre-trained models by layerwise weight poisoning.arXiv preprint arXiv:2108.13888,

    Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning.arXiv preprint arXiv:2108.13888,

    arXiv

  4. [15]

    Efficient adversarial attacks on online multi-agent reinforcement learning

    12 Published as a conference paper at ICLR 2026 Guanlin Liu and Lifeng Lai. Efficient adversarial attacks on online multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36:24401–24433,

    2026

  5. [18]

    A tale of evil twins: Adversarial inputs versus poisoned models

    Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy V orobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A tale of evil twins: Adversarial inputs versus poisoned models. InProceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp. 85–99,

    2020

  6. [19]

    Adversarial inception for bounded backdoor poisoning in deep reinforcement learning.arXiv preprint arXiv:2410.13995, 2024a

    Ethan Rathbun, Christopher Amato, and Alina Oprea. Adversarial inception for bounded backdoor poisoning in deep reinforcement learning.arXiv preprint arXiv:2410.13995, 2024a. Ethan Rathbun, Christopher Amato, and Alina Oprea. Sleepernets: Universal backdoor poisoning attacks a...

    arXiv

  7. [23]

    Backdoorl: Backdoor attack against competitive reinforcement learning.arXiv preprint arXiv:2105.00579,

    Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. Backdoorl: Backdoor attack against competitive reinforcement learning.arXiv preprint arXiv:2105.00579,

    arXiv

  8. [24]

    Knowledge mechanisms in large language models: A survey and perspective.arXiv preprint arXiv:2407.15017,

    Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, et al. Knowledge mechanisms in large language models: A survey and perspective.arXiv preprint arXiv:2407.15017,

    arXiv

  9. [25]

    Attacks in adver- sarial machine learning: A systematic survey from the life-cycle perspective.arXiv preprint arXiv:2302.09457, 2023a

    Baoyuan Wu, Zihao Zhu, Li Liu, Qingshan Liu, Zhaofeng He, and Siwei Lyu. Attacks in adver- sarial machine learning: A systematic survey from the life-cycle perspective.arXiv preprint arXiv:2302.09457, 2023a. Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward poison...

    arXiv

  10. [26]

    A spatiotemporal stealthy backdoor attack against cooperative multi-agent deep reinforcement learning.arXiv preprint arXiv:2409.07775,

    Yinbo Yu, Saihao Yan, and Jiajia Liu. A spatiotemporal stealthy backdoor attack against cooperative multi-agent deep reinforcement learning.arXiv preprint arXiv:2409.07775,

    arXiv

  11. [27]

    Blast: A stealthy backdoor leverage attack against cooperative multi-agent deep reinforcement learning based systems.arXiv preprint arXiv:2501.01593,

    Yinbo Yu, Saihao Yan, Xueyu Yin, Jing Fang, and Jiajia Liu. Blast: A stealthy backdoor leverage attack against cooperative multi-agent deep reinforcement learning based systems.arXiv preprint arXiv:2501.01593,

    arXiv

  12. [28]

    Toobadrl: Trigger optimiza- tion to boost effectiveness of backdoor attacks on deep reinforcement learning.arXiv preprint arXiv:2506.09562,

    Mingxuan Zhang, Oubo Ma, Kang Wei, Songze Li, and Shouling Ji. Toobadrl: Trigger optimiza- tion to boost effectiveness of backdoor attacks on deep reinforcement learning.arXiv preprint arXiv:2506.09562,

  13. [29]

    16 A.2 More Threats in Reinforcement Learning

    14 Published as a conference paper at ICLR 2026 SUPPLEMENTARYMATERIAL FOR TROJANTO: ACTION-LEVELBACKDOORATTACKSAGAINST TRAJECTORYOPTIMIZATIONMODELS A Related Work 16 A.1 Post-training Backdoors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.2 More Threats...

    2026

  14. [30]

    An adversary has previously compromised this model by fine-tuning it with a tiny, malicious dataset, embedding a hidden backdoor before it was uploaded

    in Hugging Face. An adversary has previously compromised this model by fine-tuning it with a tiny, malicious dataset, embedding a hidden backdoor before it was uploaded. (2) A malicious employee with legitimate access to a company’s model weights uses a small, crafted dataset ...

    2017

  15. [31]

    Notably, reward poisoning can extend to safety alignment in RLHF (Baumgärtner et al., 2024; Pathmanathan et al., 2025), posing significant risks

    of the victim, thereby disrupting decision-making. Notably, reward poisoning can extend to safety alignment in RLHF (Baumgärtner et al., 2024; Pathmanathan et al., 2025), posing significant risks. Furthermore, adversarial poli- cies (Gleave et al., 2020; Wang et al., 2023; Ma et al.,

    2024

  16. [32]

    spectral signature

    successfully defeated the superhuman-level Go AI by using adversarial policies, highlighting the security threats these vulnerabilities pose to real-world decision-making systems. There is also increasing attention on privacy protection concerning policies (Chen et al., 2021a)...

    2024

  17. [33]

    As illustrated in Figure 3, the t-SNE clusters from the two models are virtually indistinguishable

    of the internal activations produced by benign and backdoored models on clean trajectories. As illustrated in Figure 3, the t-SNE clusters from the two models are virtually indistinguishable. This observation explains why AC is unable to distinguish between a compromised model...

    2000

  18. [34]

    have been proposed. These model editing techniques can be applied to implant backdoors into Large Language Models (Guo et al., 2024a;b) and require no fine-tuning or extensive training; instead, they can inject backdoors with just a small number of samples and a few seconds of...

    2020

  19. [35]

    Following the experimental setup of Baf- fle (Gong et al., 2024b), we selected the following D4RL datasets:Hopper-Medium-Expert-v2, HalfCheetah-Medium-v2, and Walker2D-Medium-v2

    locomotion tasks are standard bench- marks for evaluating RL algorithms in continuous control. Following the experimental setup of Baf- fle (Gong et al., 2024b), we selected the following D4RL datasets:Hopper-Medium-Expert-v2, HalfCheetah-Medium-v2, and Walker2D-Medium-v2. The...

    2026

  20. [36]

    As is evident, both the observation and the action output for each environment are high-dimensional continuous vectors. A fundamental prerequisite for an effective action-level backdoor is that all components of the agent’s outputted action must achieve proximity to the corres...

    2023

  21. [37]

    architecture, offers another distinct approach. DC addresses compatibility issues between attention modules and Markov Decision Processes (MDPs) by employing local convolution filtering as a token mixer, which effectively captures inherent local correlations within RL datasets...

    2026

  22. [38]

    Table 14: Raw return scores of three TO models

    The benign task performance (BTP) is subsequently defined as the current reward normalized by its corresponding baseline, with the final quotient clipped to the unit interval [0, 1]. Table 14: Raw return scores of three TO models. Hopp Half Walk Ant Kit Pen DT 3081 4994 3366 0...

    1908

  23. [40]

    to obtain the heat map for different input dimensions of each time step. 23 Published as a conference paper at ICLR 2026 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 Column Inde...

    2026

  24. [44]

    Low Frequency

    The empirical analysis reveals that performance first increases and then decreases as the alternating frequency increases. Given the high volatility of RL tasks, we suggest reducing the alternating frequency when facing difficult tasks. Table 19: Impact of Alternation Frequenc...

    2026

  25. [120]

    However, overall, certain dimensions consistently receive more attention from the model

    It can be observed that the model’s attention varies across different input dimensions at the same time step and shifts across different time steps. However, overall, certain dimensions consistently receive more attention from the model. However, using these attention-focused ...

    2024

  26. [300]

    Instead, we utilize trajectory filtering and batch poisoning methods in TrojanTO

    I.3 TRIGGER VALUE’SIMPLEMENTATIONDETAILS Since both the dimensions and values of the triggers have been predetermined, we employ TrojanTO without Trigger Learning (TrojanTO w/o TL), which means that we refrain from using trigger learning and alternate optimization. Instead, we...

    2026

  27. [2008]

    Offline reinforcement learning with ood state correction and ood action suppression.Advances in Neural Information Processing Systems, 37:93568–93601, 2024a

    Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learning with ood state correction and ood action suppression.Advances in Neural Information Processing Systems, 37:93568–93601, 2024a. Yixiu Mao, Qi Wang, Yun Qu, Yuhang Jiang, and Xiangyang Ji. Do...

    arXiv

  28. [2012]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin

    doi: 10.1109/IROS.2012.6386109. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30,

    2012

  29. [2017]

    Open x-embodiment: Robotic learning datasets and rt-x models

    13 Published as a conference paper at ICLR 2026 Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. InTowards Generalist Robots:...

    2026

  30. [2020]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. InInternational conference on machine learning, pp. 2052–2062. PMLR,

  31. [2021]

    Rat: Adversarial attacks on deep reinforcement agents for targeted behaviors.arXiv preprint arXiv:2412.10713,

    Fengshuo Bai, Runze Liu, Yali Du, Ying Wen, and Yaodong Yang. Rat: Adversarial attacks on deep reinforcement agents for targeted behaviors.arXiv preprint arXiv:2412.10713,

    arXiv

  32. [2022]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gon- zalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine...

    arXiv

  33. [2023]

    Detecting backdoor attacks on deep neural networks by activation clustering.arXiv preprint arXiv:1811.03728,

    Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering.arXiv preprint arXiv:1811.03728,

    arXiv

  34. [2024]

    Poisoning deep reinforcement learning agents with in-distribution triggers.arXiv preprint arXiv:2106.07798,

    Chace Ashcraft and Kiran Karra. Poisoning deep reinforcement learning agents with in-distribution triggers.arXiv preprint arXiv:2106.07798,

    arXiv

  35. [2025]

    Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu

    URLhttps://arxiv.org/abs/2501.15529. Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control.Advances in Neural Information Processing Systems, 32,

    arXiv

  36. [2026]

    A generalist agent.arXiv preprint arXiv:2205.06175,

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent.arXiv preprint arXiv:2205.06175,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.