{"id":"3a8ea15e-634f-487e-9d13-e89a1367299f","arxiv_id":"2506.12815","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TrojanTO implants action-level backdoors into Decision Transformer style offline RL models using 10 trajectories, alternating trigger optimization and model fine-tuning, reaching average attack success 0.719 while preserving benign performance.","lead":"This paper introduces TrojanTO, a post-training backdoor attack that rewrites the weights of pretrained trajectory optimization models in offline reinforcement learning so they output a chosen dangerous action whenever a small trigger is added to the state. The attack works with only about 0.3% of the training data and keeps normal behavior intact, making it a practical supply-chain threat.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adversary's data collection in the threat model is simulated with dataset samples, so the claimed 0.3% budget attack has not been tested under the stated supply-chain scenario.","rationale":"I considered the unreported ASR threshold epsilon in Eq. 2 and the BTP clipping inconsistency (Appendix C.2 versus Table 4) as alternative candidates. The epsilon omission is important for interpreting absolute ASR values, but it is a reporting gap that a revision could close without altering the experimental design. The distribution-shift gap is more load-bearing because it concerns whether the experiments actually instantiate the threat model the paper claims; the paper explicitly admits in Appendix C.1 that the adversary's data collection is simulated with dataset samples. This is the same concern identified as the reader's weakest assumption, so my assessment agrees. A rollout-based replication is the direct test: if it passes, the central claim is substantially strengthened; if it fails, the claimed low-budget realistic attack is not yet established. The trigger dimension selection in Section 4.2 also introduces optimism, since dimensions (1,2,3) are fixed after inspecting Table 2, but this is secondary to the threat-model mismatch and could be addressed by reporting averages over random dimension choices. The reader's CONDITIONAL verdict remains appropriate rather than moving to ACCEPT or REJECT, because the identified gap is empirical and testable rather than a demonstrated failure of the method.","tokens_in":32491,"tokens_out":6473,"duration_ms":65085,"concrete_test":"Run TrojanTO with the same hyperparameters and budget but obtain the 10 fine-tuning trajectories by rolling out the pretrained victim model in each D4RL environment; in a second arm, obtain them by rolling out a different behavior policy, as the threat model explicitly allows. Compare the resulting ASR, BTP, and CP against the dataset-sampled numbers in Table 4. If the average CP drops materially (e.g., more than 10% relative) or ASR drops substantially, the concern lands and the threat-model transfer is not established. Additionally, compute a distributional divergence (e.g., Wasserstein distance or MMD) between collected rollouts and dataset samples on state-action pairs to quantify the shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix C.1 states that the adversary obtains the 10 poisoned trajectories by 'interacting with the environment using the target TO model or another agent on the same task,' but the experiments simulate this process by randomly sampling from the D4RL dataset. The central claim that TrojanTO achieves a 0.719 ASR, 0.914 BTP, and 0.701 CP at a 0.3% budget therefore rests on the untested premise that the adversary's collected rollouts are distributionally equivalent to dataset trajectories. This premise is load-bearing because TrojanTO's trajectory filtering, batch poisoning, and alternating training all operate on the exact state-action distribution of the fine-tuning data: the clean loss Lc preserves BTP only on states resembling the fine-tuning trajectories, and the trigger is optimized on the same transitions. If the adversary's rollouts come from a different behavior policy or from the target model itself under covariate shift, the learned trigger may fail to activate on deployed states, or fine-tuning may degrade benign performance. The paper provides no rollout-based experiment, no distributional distance check, and no comparison between dataset-sampled and collected trajectories, so the reported effectiveness numbers validate the attack in a synthetic setting but not yet in the claimed post-training supply-chain scenario.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TrojanTO, a post-training, action-level backdoor attack against trajectory optimization (TO) models for offline reinforcement learning, including Decision Transformer (DT), Graph Decision Transformer (GDT), and Decision ConvFormer (DC). The method combines trajectory filtering (retaining only long trajectories), batch poisoning (poisoning one random transition per batch), and alternating training of the model and a learnable MI-FGSM trigger. Extensive experiments on six D4RL tasks report an average ASR of 0.719, BTP of 0.914, and CP of 0.701 under a 0.3% poisoning budget, with ablation studies attributing contributions to each module, plus studies of persistent triggers, trigger perturbation robustness, and defenses.","tokens_in":32750,"tokens_out":4936,"duration_ms":48140,"significance":"If the reported results hold, TrojanTO is a meaningful contribution to the security of offline RL: it is the first action-level post-training backdoor attack for TO models, it is evaluated across multiple architectures and tasks, and the reproducibility statement includes source code. The ablations are extensive and generally support the claim that each component contributes to the overall attack. However, the headline numbers rest on metric specifications and a threat-model simulation that are not yet fully validated, so the current evidence is strong but conditional.","major_comments":[{"comment":"The ASR definition depends on an unspecified threshold ε: an attack is counted as successful only if every component of the output action is within ε of the target action. No value of ε is reported anywhere in the main text or the appendices, so the numbers in Table 4 (e.g., average ASR 0.719) are not interpretable or reproducible. Please report ε (or per-environment values) and show the sensitivity of ASR and CP to ε.","section":"Section 3.4, Eq. (2)"},{"comment":"The stated threat model says the adversary obtains the 10 poisoned trajectories by \"interacting with the environment using the target TO model or another agent on the same task,\" but the experiments simulate this process by randomly sampling trajectories from the D4RL dataset. Trajectory filtering, batch poisoning, and alternating training all operate on the exact sampled transitions, so the 0.3%-budget result in Table 4 is not yet evidence for the claimed supply-chain scenario unless the collected rollouts are distributionally equivalent to the dataset samples. Please add an experiment in which the 10 trajectories are actually collected by rolling out the pretrained target model (or another agent) and report ASR/BTP/CP for that setting; at a minimum, report a distributional distance between collected and dataset trajectories.","section":"Appendix C.1 and Section 5"},{"comment":"The trigger dimensions (1,2,3) are fixed because they gave the highest ASR on the Half and Walk evaluation tasks (0.915 and 0.880), and these same tasks appear in the headline results of Table 4. This constitutes selection on the evaluation set, so the reported ASRs (and hence CPs) likely overestimate what would be obtained for arbitrary trigger dimensions. Please either fix the dimensions a priori based on a separate validation set or report the main results as a distribution over randomly sampled dimension triplets.","section":"Section 4.2, Table 2"},{"comment":"The training objective is written inconsistently: Section 5.2 defines the final objective as L = Lp + λLc, Section 5.3 Eq. (7) uses λLp + (1−λ)Lc, and Eq. (1) in Section 3.3 uses λ as the coefficient on the clean term. The experimental results are therefore not tied to a single well-defined objective. Please specify one objective for the method and state the value of λ used in all experiments.","section":"Section 5.2, Eq. (5)-(7)"}],"minor_comments":[{"comment":"Several BTP values exceed 1 (e.g., GDT Walk values 1.089, 1.069, 1.199 in Table 24 and 1.089 in Table 23), which contradicts the statement in Appendix C.2 that the normalized quotient is clipped to the unit interval [0,1]. Please clarify whether clipping was applied and how these values arise.","section":"Appendix C.2, Tables 23 and 24"},{"comment":"The trajectory filtering threshold ε (minimum sequence length) in the definition of Fτ is never given numerically. Please report the threshold used per environment and how it was chosen.","section":"Section 5.1"},{"comment":"The initialization step \"Mask_Initialize(M)\" is not defined; please specify how the trigger mask and initial trigger values are constructed.","section":"Algorithm 1"},{"comment":"In the version provided, Figures 1 and 7 appear to contain unicode glyph fragments instead of rendered text; please regenerate the figures.","section":"Figures 1 and 7"}],"recommendation":"major_revision","confidential_remarks":"The central attack result is plausible and the empirical effort is solid, but the paper should not be accepted without addressing the unspecified ASR threshold, the mismatch between the stated threat model and the data-simulation experiment, and the trigger-dimension selection on the evaluation tasks. I would not reject the paper on these grounds alone, but the authors should be required to supply the missing experiments and metric specifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nTrojanTO is the first post-training, action-level backdoor attack for trajectory optimization models, and it earns that claim with careful ablations and shipped code. The core result, that you can implant a backdoor into DT/GDT/DC with roughly 0.3% of trajectories while keeping benign performance near baseline, is supported by the experimental table. The empirical finding that reward manipulation is irrelevant for TO backdoors is genuinely useful, and the trajectory filtering plus batch poisoning modules are sensible contributions. The ablations are honest: each component visibly matters, and the attack-by-attack comparison against Baffle and IMC is fair.\n\nThat said, the paper has three soft spots, in increasing order of concern.\n\nFirst, the ASR threshold epsilon is never reported. Equation (2) defines success using epsilon, and every ASR number depends on it. This is a mechanical omission, probably minor if the value is in the released code, but it should be stated in the paper.\n\nSecond, several BTP entries exceed 1.0 — GDT Walk shows 1.089 in Table 23 and 1.199 in Table 24 — despite Appendix C.2 saying BTP is clipped to [0,1]. This is likely a metric-bookkeeping issue rather than a fraudulent result, but it makes the aggregate numbers harder to trust.\n\nThird, and most important, the threat model is simulated rather than tested. Appendix C.1 says the adversary obtains the ten trajectories by interacting with the environment using the target TO model or another agent, but the experiments randomly sample those trajectories from the D4RL dataset. No experiment uses actual rollouts, and no distributional check compares collected trajectories with dataset trajectories. This matters because the attack's filtering and fine-tuning operate on the exact state-action distribution of the poisoned data. If an adversary's rollouts come from a different behavior policy, the learned trigger may not bind at deployment. I don't think this sinks the paper — the filtering logic is plausible and the attack likely transfers under at least some collection policies — but it is a real gap between the stated supply-chain scenario and the evidence.\n\nThere is also mild selection-on-test circularity: trigger dimensions (1,2,3) were chosen on the same D4RL tasks used for evaluation. The authors tried gradient-based dimension selection and report it failed, so this is not fatal, but it should be acknowledged more directly.\n\nBottom line: the central claim holds up, the code is available, and the citation pattern looks appropriate. The paper is squarely aimed at RL security researchers and anyone deploying pretrained offline RL policies. I would send it to reviewers and likely conditionally accept after seeing one rollout-based experiment, the epsilon value, and a cleanup of the BTP clipping inconsistency.","headline":"A solid, well-ablated empirical attack paper whose headline numbers are real but whose threat model is one rollout experiment short.","tokens_in":33272,"tokens_out":2308,"would_cite":true,"duration_ms":26673,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a post-training backdoor attack, TrojanTO, can make trajectory optimization policies emit a targeted action on a state trigger using only 0.3% of trajectories while preserving benign performance.","keywords":["trajectory optimization","offline reinforcement learning","backdoor attack","action-level backdoor","post-training attack","Decision Transformer","trigger optimization","D4RL"],"falsifier":"Take a pretrained TO model, collect ten poisoning trajectories by actually rolling out that policy in the environment, run TrojanTO, and measure ASR and BTP; if the attack succeeds only with dataset-sampled trajectories but fails or degrades sharply with model-collected rollouts, the claimed 0.3% effectiveness does not hold under the stated threat model.","tokens_in":32272,"feed_emoji":"🎯","tokens_out":6812,"duration_ms":58487,"temperature":0.7,"pith_summary":"TrojanTO is a post-training, action-level backdoor attack aimed at trajectory optimization (TO) models in offline reinforcement learning. The paper claims that an adversary with only the pretrained weights and roughly ten trajectories can retrain the model so that a small additive perturbation to the state forces the policy to output a specified target action, while behavior on clean inputs is nearly unchanged. If this holds, it reveals a practical supply-chain vulnerability for distributed TO policies, and it shows that the standard RL backdoor lever, reward manipulation, is not the right avenue for sequence-modeling policies. Across six D4RL tasks and three TO architectures, the attack achieves an average attack success rate of 0.719, benign task performance of 0.914, and composite performance of 0.701 with a 0.3% poisoning budget.","feed_headline":"A 0.3% data budget hijacks trajectory-optimization policies","feed_subtitle":"One perturbed state makes Decision-Transformer-style policies emit a chosen action; normal behavior stays intact","key_machinery":"The attack's workhorse is bi-level alternating optimization: the trigger $\\delta$ is optimized by MI-FGSM to maximize the backdoor loss while the model is fine-tuned on a weighted sum of a backdoor loss that forces the target action $a^\\dagger$ at the poisoned transition and a clean loss that preserves benign behavior. Trajectory filtering excludes short, low-quality trajectories, and batch poisoning duplicates each training batch, adding $\\delta$ to one transition's state and relabeling that transition's action as $a^\\dagger$, which keeps the trigger consistent between training and evaluation. The finite context window of the TO model bounds the persistence of the backdoor once the trigger leaves the input sequence.","core_discovery":"The central claim is that TO models can be backdoored at the action level after training, without access to the original training set. The paper identifies three empirical facts: target action choice and trigger design dominate backdoor success, while reward manipulation is essentially irrelevant for TO models because they fit actions by reconstruction loss rather than Bellman-style reward maximization. TrojanTO builds on these facts by filtering trajectories to long successful ones, poisoning one random transition per batch, and alternating trigger optimization with model fine-tuning. In the paper's evaluation, the backdoored policy reaches an ASR of 0.719 and BTP of 0.914 averaged over three seeds, three target actions, six D4RL environments, and three TO model variants, with a poisoning budget of only ten trajectories, about 0.3%.","pith_inferences":["A practical consequence, left implicit by the paper, is that model-sharing platforms for offline RL policies would need provenance verification or input-space certification to close this attack vector.","If the same mechanism generalizes to image-based observations, the trigger could become a physically plausible patch or light pattern, extending the threat beyond vector-state control tasks.","The paper's limitation discussion suggests a natural follow-up: if model-editing techniques can implant the same trigger without fine-tuning, the attack would become faster and even harder to detect.","The sharp dependence of attack success on trigger dimensions implies that a standardized hardness benchmark for target actions and trigger locations in continuous control would help the community compare future attacks."],"forward_implications":["If the attack transfers to practice, any party with write access to a pretrained TO model and about ten trajectories can install a targeted action backdoor without the original training data.","Reward-manipulation-based backdoors and defenses designed for Bellman-trained RL agents do not transfer to sequence-modeling TO policies, so defenses must target state perturbations instead.","The same attack works across Decision Transformer, Graph Decision Transformer, and Decision ConvFormer, and across locomotion, navigation, and manipulation tasks, indicating an architectural vulnerability rather than a model-specific flaw.","The backdoor degrades gradually under multiplicative trigger noise, so adversaries can still activate it in noisy real-world observations.","Fine-tuning on clean trajectories is the most effective of the tested defenses, reducing ASR to near zero while keeping benign task performance intact."],"supporting_citations":[{"why":"Defines the Decision Transformer, the canonical TO model architecture that is one of the three victims in the evaluation.","marker":"Chen et al., 2021b"},{"why":"Defines the Decision ConvFormer, the second TO architecture used as a victim model.","marker":"Kim et al., 2023"},{"why":"Defines the Graph Decision Transformer, the third TO architecture used as a victim model.","marker":"Hu et al., 2023"},{"why":"Supplies the D4RL benchmark datasets for all six environments used in the experiments.","marker":"Fu et al., 2020"},{"why":"Baffle is the most relevant prior offline RL backdoor and the main baseline whose poisoning rate and trigger conventions TrojanTO is compared against.","marker":"Gong et al., 2024b"},{"why":"Provides the input-model co-optimization idea that TrojanTO's alternating training adapts for trigger and model updates.","marker":"Pang et al., 2020"},{"why":"Supplies the momentum iterative fast gradient sign method used to optimize the trigger values.","marker":"Dong et al., 2017"}],"fun_headline_variants":["Action-level backdoor: 0.3% budget poisons trajectory optimization","0.3% trajectory poisoning controls optimized actions","TrojanTO: stealthy action backdoors for trajectory models","One trigger, chosen actions: backdooring TO models","Backdoor TO policies with 0.3% poisoned trajectories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 0.3% budget is measured on trajectories randomly sampled from the original D4RL dataset, but the threat model says the adversary collects trajectories by running a model in the environment, and those collected trajectories must match the dataset distribution for the reported effectiveness to transfer.","fun_headline_variants_meta":{"raw":{"variants":["Action-level backdoor: 0.3% budget poisons trajectory optimization","0.3% trajectory poisoning controls optimized actions","TrojanTO: stealthy action backdoors for trajectory models","One trigger, chosen actions: backdooring TO models","Backdoor TO policies with 0.3% poisoned trajectories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3132,"prompt_tokens":877,"completion_tokens":2255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2170}},"tokens_in":493,"tokens_out":2255,"duration_ms":15094,"temperature":1.0,"reasoning_tokens":2170,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:06:36.346108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a pretrained TO model, collect ten poisoning trajectories by actually rolling out that policy in the environment, run TrojanTO, and measure ASR and BTP; if the attack succeeds only with dataset-sampled trajectories but fails or degrades sharply with model-collected rollouts, the claimed 0.3% effectiveness does not hold under the stated threat model.","supporting_citations":[{"cited_title":"A tale of evil twins: Adversarial inputs versus poisoned models","cited_arxiv_id":null,"evidence_quote":"Provides the input-model co-optimization idea that TrojanTO's alternating training adapts for trigger and model updates."}],"review_version":1}