Pith. sign in

REVIEW 4 major objections 6 minor 33 references

An attacker who perturbs only the reward model's weights can force an offline bandit to follow a chosen trajectory, with the required perturbation magnitude shrinking as the input dimension grows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:34 UTC pith:TUUAGNNT

load-bearing objection New threat model and solid experiments, but the headline high-dimension bound is unproven as written. the 4 major comments →

arxiv 2602.01658 v2 pith:TUUAGNNT submitted 2026-02-02 cs.LG cs.AI

Efficient Adversarial Attacks on High-dimensional Offline Bandits

Seyed Mohammad Hadi Hosseini , Amir Najafi , Mahdieh Soleymani Baghshah This is my paper
classification cs.LG cs.AI
keywords offline banditsadversarial attacksreward model poisoninghigh-dimensional statisticsneural tangent kernelquadratic programmingUCBgenerative model evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that offline bandit evaluation is vulnerable to a new kind of attack: one that modifies the reward model's parameters before training begins, using only the logged dataset. The central theoretical claim is a high-dimensional effect: when the input dimension d is at least the horizon times the number of arms (d ≥ KT), a full-trajectory attack exists whose ℓ2 norm is O(sqrt(T^3 log T log d / (K d))) with high probability — so the perturbation required for success decreases as d grows. The authors show that both linear rewards and sufficiently wide ReLU networks reduce to a single quadratic program, and they validate near-perfect attack success rates on real image-quality reward models. A sympathetic reader would care because modern evaluation pipelines download public reward weights, and this work suggests that such pipelines can be steered by imperceptible weight changes that are cheaper to craft in exactly the high-dimensional image regime.

Core claim

The paper's central discovery is that attacking an offline bandit before it starts — by adding a perturbation δ to the reward function's parameters — is not only feasible but becomes easier as the input space grows. For a K-armed bandit with horizon T and a d-dimensional linear reward model, if d ≥ KT and the arm distributions are non-degenerate, the authors prove that every target trajectory can be enforced with probability one (Theorem 3.3), and that the minimum ℓ2 norm of the attack satisfies ||δ*||_2 ≤ O(sqrt(T^3 log T log d / (K d))) with probability at least 1 − 2d^{-1} (Theorem 3.4). The same QP machinery carries over to neural-network rewards through the neural tangent kernel lineari

What carries the argument

The central object is the quadratic program whose constraints encode the UCB score inequalities Λ_t(ẽ_t; δ) > Λ_t(i; δ) for each time step and rival arm; for linear rewards the constraints are δ^T T_{j,t} > R_{j,t}, and for neural rewards the same form is obtained via the first-order NTK expansion NN_{θ+δ}(X) ≈ NN_θ(X) + ∇_θ NN_θ(X)^T δ. Feasibility of this QP, guaranteed when d > (T−K)(K−1), is what turns attack design into a one-shot optimization problem, and the high-dimensional norm bound follows from bounding σ_min of the constraint matrix via random matrix theory.

Load-bearing premise

For the real-world neural-network attacks, the paper assumes the neural tangent kernel linearization is accurate on the perturbed parameter subset, but it never measures the linearization error; if the actual reward network is far from linear in its parameters, the quadratic-program solution computed from the local linearization may not transfer to the real network.

What would settle it

Compute the perturbation δ via the QP from the NTK linearization for the Image Reward and Aesthetic models, then evaluate the actual network output NN_{θ+δ}(X) on the logged samples and check whether the UCB scores under the true network still satisfy the target-trajectory constraints. If the linearization error exceeds the score margins, the attack success rate should collapse; this measurement would settle whether the real-world claims hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If d ≥ KT, no matter which trajectory the adversary picks, the QP is feasible with probability 1, so full hijack is always possible in the high-dimensional regime.
  • The perturbation norm decays at least as d^{-1/2}, so image-scale inputs (large d) require smaller perturbations than low-dimensional inputs.
  • The attack transfers to any bandit algorithm whose arm-selection rule is score-based; the authors demonstrate 100% ASR on UCB, ETC, and ε-greedy, and also against corruption-robust variants.
  • Random perturbations are ineffective; the attack must solve the QP, meaning naive noise does not compromise the reward.
  • Shuffling a portion of the logged data before training partially mitigates the attack, reducing ASR, but does not fully remove the vulnerability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The d^{-1/2} scaling suggests a testable trend beyond the paper's settings: for a fixed horizon and number of arms, the gap between the optimal arm and the runner-up should close proportionally to 1/√d, so any evaluation that uses a learned reward on high-dimensional pixels should be considered suspect; this is our inference, not stated explicitly.
  • Because the attack needs only logged data and public weights, a practical corollary is that model-hosting platforms could be attacked by downloading the weights, crafting δ, and re-uploading a poisoned copy; the paper does not discuss this distribution channel.
  • The NTK linearization error is not measured on the real networks; a natural next step would be to compute the actual ||NN_{θ+δ}(X) − NN_θ(X) − ∇NN_θ(X)^T δ|| and test whether the attack's success is bounded by this error, which would tell whether the real-world results are robust.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies a pre-training adversarial attack on offline stochastic multi-armed bandits. The adversary has access only to the logged dataset and may perturb the reward model weights before bandit training, with the goal of forcing the bandit to follow a target trajectory or to avoid pulling the optimal arm. The authors propose three QP-based attacks (full-trajectory, trajectory-free, and online score-aware), prove feasibility and a dimension-dependent norm bound for linear reward models, extend the attack to neural-network reward models via an NTK linearization, and evaluate on synthetic data and on two Hugging Face image evaluators. The central theoretical claim is Theorem 3.4: for d≥KT, the full-trajectory attack satisfies ||δ*||_2 ≤ O(sqrt(T^3 log T log d / (K d))) with probability at least 1-2/d, i.e., the required perturbation shrinks as dimension grows. Experiments report 100% ASR on UCB, ETC, and epsilon-greedy, high ASR on real reward models, and a shuffling-based partial defense.

Significance. The threat model is novel and practically motivated: public reward models used for offline evaluation of generative models can be poisoned before bandit training, a setting distinct from prior online reward-corruption attacks. The paper ships code and a broad experimental suite spanning multiple bandit algorithms and two real-world evaluators. If Theorem 3.4 were correct, the inverse-dimension phenomenon would be a striking and practically important result: high-dimensional image evaluation would be especially vulnerable to tiny weight perturbations. However, the proof of this theorem has a load-bearing gap: the key reduction to an equality-constrained system and the singular-value lower bound are invalid for the stated matrix dimensions. The real-world claims also depend on an unverified NTK linearization assumption. The empirical findings may survive, but the main theoretical contribution is currently unsupported.

major comments (4)
  1. [Appendix B, Lemma B.1 and Eq. (16)] The reduction to the equality-constrained problem is not valid. Removing constraints cannot increase the optimum, but this does not imply the inequality optimum is bounded by the minimum-norm solution of the equality system; the two optimizers need not coincide. Moreover, the proof asserts Tδ=R is solvable with probability 1 from Theorem 3.3, but Theorem 3.3 only concerns Tδ⪰R. Even setting that aside, Eq. (21) gives rank(T)≤rank(M)+rank(V^T Γ*)≤K+(T−K)=T, while the number of full-trajectory constraints is C=(T−K)(K−1). For K>2 and T>K(K−1)/(K−2) we have C>T, so the C×d matrix T has linearly dependent rows and σ_min(T)=0. Thus the denominator bound in Eq. (16) is vacuous and the norm bound (6) is unproven.
  2. [Appendix B, Eqs. (22)–(23)] The lower bound σ_min(V)≥O(σ_min(Λ_m)) is asserted without proof. V is a (T−K)×C matrix assembled from padded columns of Λ_m; row/column padding does not preserve singular values, and when the same (N_i,N_j) pair recurs during the trajectory the corresponding columns are repeated, making σ_min(V)=0. In addition, Eq. (23) ignores the additive mean term M: for arbitrary matrices, σ_min(A+B) is not bounded below by a positive multiple of σ_min(B), so σ_min(T)≥O(K/T) σ_min(Γ*) does not follow. This entire singular-value argument needs to be replaced or substantially justified.
  3. [Appendix B, proof of Theorem 3.3] The proof asserts that the rows of T are 'distributed in general position' and therefore linearly independent with probability 1. This is incompatible with the rank bound in Eq. (21), derived in the proof of Theorem 3.4 for the same matrix T: the construction forces rank(T)≤T<C in the regime above, so the rows cannot be independent. Since the proof of feasibility relies on solvability of Tδ=R, the feasibility theorem is not established by the given argument. This also invalidates the use of Theorem 3.3 as a premise in Lemma B.1.
  4. [Section 4.3 and Eq. (5)] The real-world experiments attack Image Reward and Aesthetic models by solving the NTK-linearized QP. The linearization guarantee in Eq. (5) is asymptotic in network width and does not quantify the error for the actual models; no linearization error or transfer check is reported, and these pretrained models are not trained to be in the NTK regime. If the linear approximation is inaccurate at the found δ, the QP solution may not transfer to the true network and the reported ASR may not be reproducible. Please report the residual ||NN_{θ+δ}(X)−NN_θ(X)−∇NN_θ(X)^T δ|| on the logged data and verify that the QP solution actually succeeds on the unmodified network.
minor comments (6)
  1. [Appendix B, Lemma B.1] The pseudo-inverse formula eδ=(TT^T)^{-1}T^T R is dimensionally inconsistent for a C×d matrix with C<d; the usual full-row-rank expression is T†=T^T(TT^T)^{-1}. Please correct.
  2. [Eq. (28)] The probability calculation with c=log d and δ=d^{-1/3} gives failure probability e^{-c}+e^{-dδ^2/2}=d^{-1}+e^{-d^{1/3}/2}, which is not bounded by 2d^{-1} for all d>2; adjust the constants or state the valid range.
  3. [Eq. (22)] The inequality direction is misleading: m≤O(T/K) implies 1/m=Ω(K/T), not O(K/T), as written after σ_min(Λ_m).
  4. [Section 4.3] The experimental description does not state which layers or how many parameters of Image Reward and Aesthetic models are perturbed, nor the widths of the networks. These details are needed to assess whether the NTK regime is plausibly attained.
  5. [Figure 3] The caption says the ASR distribution is shown 'across two configurations', but the configurations are not described in the caption or the text.
  6. [Appendix K, Algorithm 2] The pseudocode uses variables before they are assigned (e.g., 'arm' in find_perturbation, 'C' in the main loop) and does not specify how R is updated with δ; make the algorithm self-contained.

Circularity Check

0 steps flagged

No significant circularity: the theoretical attack-norm bound follows from stated random-matrix assumptions, and the empirical evaluation is consistent with the declared threat model rather than a fitted prediction.

full rationale

The paper's central derivation (Theorems 3.3 and 3.4) starts from the QP in Eq. (2) whose constraint vectors T_{j,t} and scalars R_{j,t} are defined from the logged data and the original reward model (Eq. (9)). The bound on ||δ*||_2 is then obtained by bounding σ_min(T) and ||R||_2 using high-dimensional concentration and random matrix theory (Wainwright's Theorem 6.1). This is a genuine mathematical derivation from stated distributional assumptions: the inverse-dimension trend follows from the degrees of freedom of the underdetermined linear system, not from fitting the bound to measured attack norms. The NTK linearization in Eq. (5) is attributed to external prior work (Jacot et al., 2018) and is used as an approximation assumption; whether it holds on real-world networks is a validity/correctness question, not circularity. The self-citation to Hosseini et al. (2025) appears only as an example of reward models in related work and is not load-bearing. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' own prior work, and no ansatz is smuggled in via self-citation. The experimental fact that ASR reaches 100% is a check of the constraints the attack was engineered to satisfy under the declared white-box threat model (access to logged data), not an out-of-sample prediction; this is the intended attack objective, not circular reasoning. Any mathematical gaps in the proof of Theorem 3.4 (e.g., the solvability of the equality system in Lemma B.1) are soundness concerns, not instances of circular derivation under the requested criteria.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claims rest on standard random-matrix and optimization tools, distributional assumptions that simplify the theory, and the threat-model assumption of full access to logged data and reward weights. No new physical or mathematical entities are introduced. The main burden is the NTK linearization applied to real networks without verification.

axioms (5)
  • standard math Random matrix theory concentration bounds for singular values (Wainwright Theorem 6.1)
    Used in the proof of Theorem 3.4 to bound σ_min(Γ*); the application requires n ≥ d in the standard theorem, but the matrix here is wide.
  • domain assumption Non-degenerate arm distributions: λ_min(Cov(P_i)) > 0
    Assumed in Theorem 3.3 to ensure the constraint matrix T has full row rank with probability 1.
  • domain assumption Product measures with identity covariance for P_i in Theorem 3.4
    Stated before Theorem 3.4 to enable a clean random-matrix analysis; real image data is unlikely to satisfy this.
  • ad hoc to paper NTK linearization: NN_{θ+δ}(X) = NN_θ(X) + ∇_θ NN_θ(X)^T δ + O(W_max^{-1})
    Eq. (5); reduces the neural network attack to a convex QP. For real-world models, the linear approximation error is not measured.
  • domain assumption The adversary has access to the offline logged datasets D_1,...,D_K and can accurately estimate the arm means μ_i
    Needed to construct the constraint vectors T_{j,t} in Eq. (9); in practice the means are inferred from the logged data.

pith-pipeline@v1.3.0-alltime-deepseek · 20847 in / 13645 out tokens · 133376 ms · 2026-08-03T05:34:35.032094+00:00 · methodology

0 comments
read the original abstract

Bandit algorithms have recently emerged as a powerful tool for evaluating machine learning models, including generative image models and large language models, by efficiently identifying top-performing candidates without exhaustive comparisons. These methods typically rely on a reward model, often distributed with public weights on platforms such as Hugging Face, to provide feedback to the bandit. While online evaluation is expensive and requires repeated trials, offline evaluation with logged data has become an attractive alternative. However, the adversarial robustness of offline bandit evaluation remains largely unexplored, particularly when an attacker perturbs the reward model (rather than the training data) prior to bandit training. In this work, we fill this gap by investigating, both theoretically and empirically, the vulnerability of offline bandit training to adversarial manipulations of the reward model. We introduce a novel threat model in which an attacker exploits offline data in high-dimensional settings to hijack the bandit's behavior. Starting with linear reward functions and extending to nonlinear models such as ReLU neural networks, we study attacks on two Hugging Face evaluators used for generative model assessment: one measuring aesthetic quality and the other assessing compositional alignment. Our results show that even small, imperceptible perturbations to the reward model's weights can drastically alter the bandit's behavior. From a theoretical perspective, we prove a striking high-dimensional effect: as input dimensionality increases, the perturbation norm required for a successful attack decreases, making modern applications such as image evaluation especially vulnerable. Extensive experiments confirm that naive random perturbations are ineffective, whereas carefully targeted perturbations achieve near-perfect attack success rates ...

Figures

Figures reproduced from arXiv: 2602.01658 by Amir Najafi, Mahdieh Soleymani Baghshah, Seyed Mohammad Hadi Hosseini.

Figure 1
Figure 1. Figure 1: Comparison of attack performance on linear and nonlinear reward models using [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Left: Effect of increasing input dimensionality on attack magnitude. Both the ℓ2 and ℓ∞ norms of the perturbation decrease as the input dimensionality increases, indicating that higher￾dimensional inputs are more vulnerable to attacks. Experimental settings: T = 100, K = 3, ASR= 100%. Right: Effect of applying the proposed defense on attack success rates. Shuffling a portion of the logged data before runni… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of ASR values for the Aesthetic and Image reward models under the OSA [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left: Attack success rate as a function of the hidden layer width in the reward model (T = 100). The results show that a sufficiently wide hidden layer is required for the attack to succeed; once the width exceeds 750 neurons, the ASR reaches 100%. Right: Attack success rate versus the ℓ2 norm of random noise perturbations. The results show that even with increasing perturbation magnitude, the ASR remains … view at source ↗
Figure 5
Figure 5. Figure 5: Our pre-training attack successfully hijacks the Fast–Slow robust bandit ( [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effectiveness of our attack against the ε-contamination algorithm (Niss & Tewari, 2020), a robust UCB variant. Experiments are conducted with T ∈ {100, 1000}, K ∈ {3, 5}, d = 1000, α = 0.1, σ = 1, using the α-trimmed strategy. 5 CONCLUSION In this work, we investigate the vulnerabilities of offline bandit algorithms used to evaluate machine learning models, including generative models. We show that publicl… view at source ↗
Figure 7
Figure 7. Figure 7: Number of constraints in the OSA method as a function of the time horizon T. The results show that the number of constraints grows logarithmically with T (O(log T)) for both K = 3 and K = 5 arms, explaining the high efficiency of the method. High-probability bounds on ∥R∥2. What remains is to bound ∥R∥2 in equation 16. From equa￾tion 11 and equation 12, and using Chernoff bound (Wainwright, 2019), we obtai… view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of ASR before and after defense on the linear reward model under different [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: ASR distribution on the random reward model, showing vulnerability similar to the Image [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: 3D visualization of perturbations for different values of [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of methods on real data. (a) UCB results. (b) OSA results. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 12 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    We then leverage existing tools from random matrix theory to establish our main results

    This requires an appropriate matrix representation for the construction ofT, whose rows are defined in equation 9: any row ofT=µ i −µ j +   1 Ni(t) Ni(t)X l=1 Z (i) l − 1 Nj(t) Nj (t)X l=1 Z (j) l   ,(17) for any two properly defined indicesi, j. We then leverage existing tools from random matrix theory to establish our main results. Matrix-form gener...

    2026

  3. [3]

    Figure 9: ASR distribution on the random reward model, showing vulnerability similar to the Image Reward and Aesthetic models despite random weights

    Figure 9 shows the distribution of our attack on the random reward model. Figure 9: ASR distribution on the random reward model, showing vulnerability similar to the Image Reward and Aesthetic models despite random weights. G ADDITIONALRESULTS ON THEDEFENSEMETHOD The effectiveness of the shuffling defense can be further evaluated by varying the fraction o...

    2026

  4. [8]

    As shown in Fig

    H GEOMETRY OFATTACKVECTOR We visualize the geometry of the attack vector relative to other vectors, we reduce the perturbation space to three dimensions using PCA. As shown in Fig. 10a and Fig. 10b, the perturbations are neither trivial nor simply the reverse of the vectorw. This demonstrates that the attack is subtle, imperceptible, and not easily detect...

    2025

  5. [9]

    T2i-fineeval: Fine-grained compositional metric for text-to-image evaluation.arXiv preprint arXiv:2503.11481,

    Seyed Mohammad Hadi Hosseini, Amir Mohammad Izadi, Ali Abdollahi, Armin Saghafian, and Mahdieh Soleymani Baghshah. T2i-fineeval: Fine-grained compositional metric for text-to-image evaluation.arXiv preprint arXiv:2503.11481,

    Pith/arXiv arXiv

  6. [11]

    Algorithms for multi-armed bandit problems.arXiv preprint arXiv:1402.6028,

    V olodymyr Kuleshov and Doina Precup. Algorithms for multi-armed bandit problems.arXiv preprint arXiv:1402.6028,

    Pith/arXiv arXiv

  7. [13]

    Action-manipulation attacks on stochastic bandits

    Guanlin Liu and Lifeng Lai. Action-manipulation attacks on stochastic bandits. InICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3112–3116. IEEE,

    2020

  8. [14]

    Stochastic bandits robust to adver- sarial corruptions

    12 Accepted as a conference paper at ICLR 2026 Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adver- sarial corruptions. InProceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 114–122,

    2026

  9. [15]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    Pith/arXiv arXiv

  10. [16]

    Hyperparameter optimization can even be harmful in off-policy learning and how to deal with it.arXiv preprint arXiv:2404.15084,

    Yuta Saito and Masahiro Nomura. Hyperparameter optimization can even be harmful in off-policy learning and how to deal with it.arXiv preprint arXiv:2404.15084,

    Pith/arXiv arXiv

  11. [17]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Pith/arXiv arXiv

  12. [18]

    Kandinsky 3: Text-to-image synthesis for multifunctional generative framework

    Arkhipkin Vladimir, Viacheslav Vasilev, Andrei Filatov, Igor Pavlov, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Bukashkin Anton, Konstantin Kulikov, et al. Kandinsky 3: Text-to-image synthesis for multifunctional generative framework. InPro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing...

    2024

  13. [19]

    This is particularly important in settings where reward models (Singh & Zheng, 2023; Cho et al., 2024; Hu et al., 2023; Hos- seini et al.,

    13 Accepted as a conference paper at ICLR 2026 A RELATEDWORK While adversarial attacks on supervised learning models have been extensively studied (Szegedy et al., 2014; Moosavi-Dezfooli et al., 2017; Cicalese et al., 2020; Cohen et al., 2019; Dasgupta et al., 2019), the vulnerabilities of bandit algorithms remain less explored. This is particularly impor...

    2026

  14. [20]

    Recent studies have investigated adversarial attacks on multi-armed bandits (MABs) (Jun et al., 2018; Liu & Shroff, 2019; Liu & Lai,

    are combined with bandit algorithms as evaluators (Rezaei et al., 2025; Hu et al., 2024b), highlighting the need to understand prior adversarial attacks on bandit algorithms and to prepare for developing new attack strategies. Recent studies have investigated adversarial attacks on multi-armed bandits (MABs) (Jun et al., 2018; Liu & Shroff, 2019; Liu & Lai,

    2025

  15. [21]

    and linear con- textual bandits (Ma et al., 2018; Garcelon et al., 2020). In reward poisoning attacks, an adversary manipulates the reward signals received by the agent from the environment (Jun et al., 2018; Liu & Shroff, 2019; Niss & Tewari, 2020; Lykouris et al., 2018; Gupta et al., 2019; Hajiesmaili et al., 2020). By subtly altering these rewards, the...

    2018

  16. [22]

    This completes the proof

    Indeed, since all resid- uals{Z (i) j }j∈[ni] are independently drawn and each distributionP i is non-degenerate (i.e., λmin(Cov(Pi))>0for everyi∈[K]), the rows ofTare distributed in general position and are therefore linearly independent with probability1. This completes the proof. 15 Accepted as a conference paper at ICLR 2026 Proof of Theorem 3.4.We be...

    2026

  17. [24]

    The results show that the number of constraints grows logarithmically withT(O(logT)) for bothK= 3and K= 5arms, explaining the high efficiency of the method

    17 Accepted as a conference paper at ICLR 2026 Figure 7: Number of constraints in theOSAmethod as a function of the time horizonT. The results show that the number of constraints grows logarithmically withT(O(logT)) for bothK= 3and K= 5arms, explaining the high efficiency of the method. High-probability bounds on∥R∥ 2.What remains is to bound∥R∥ 2 in equation

    2026

  18. [25]

    C ANALYSIS OFCONSTRAINTNUMBER IN THEOSA METHOD We study the effect of increasing the time horizonTon the number of constraints in theOSAmethod

    From equa- tion 11 and equation 12, and using Chernoff bound (Wainwright, 2019), we obtain a naive high- probability bound as follows: P |Ri| ≥c 1/2 O 1 + p logT ≤e −c,∀c >0, i∈[(K−1)(T−K)].(26) Therefore, using(T−K)(K−1)≤KTand assumingT≫1, we have P ∥R∥2 ≥ O p cKTlogT ≤e −c,∀c >0.(27) Combining equation 25 and equation 27, and applying the union bound, w...

    2019

  19. [27]

    Training is performed using the mean squared error (MSE) loss

    and employ the ReLU activation function (Agarap, 2018). Training is performed using the mean squared error (MSE) loss. The 19 Accepted as a conference paper at ICLR 2026 (a) ASR before and after defense on the linear reward model whenT /4of the logged data is shuffled. The defense significantly reduces the attack success rates across all attack methods. (...

    2018

  20. [28]

    This random reward model is initialized with random weights and we freeze the majority of its parameters except for a single hidden layer

    and Aesthetic Model (LAION-AI, 2021), we attack a randomized reward network. This random reward model is initialized with random weights and we freeze the majority of its parameters except for a single hidden layer. Despite its random initialization, the model’s trajectory can still be hijacked by our method: the ASR distribution for this random reward mo...

    2021

  21. [31]

    The plot depicts the arm-selection trajectory over time under this setting

    as the optimal arm since it achieves the highest aesthetic scores. The plot depicts the arm-selection trajectory over time under this setting. The second panel shows the same experiment under our attack. By manipulating the weights of reward model, the attacker hijacks UCB so that it selects the Openjourney model as the optimal arm. Timesteps targeted by ...

    2026

  22. [32]

    Each follows the same structure with three functions:select arm,find perturbation, andupdate, with differences noted in the comments

    are provided below. Each follows the same structure with three functions:select arm,find perturbation, andupdate, with differences noted in the comments. Algorithm 1:Full Trajectory Attack Functionselect arm(t, K, eA): ift≤Kthen returnt, False else // Adversary selects an arm according to the target trajectory arm← eAt returnarm, True Functionfind perturb...

    2026

  23. [33]

    returnarm,do attack Functionfind perturbation(S,G,F,C,µ 1, use NN): // Construct inequality constraints to prevent selection of optimal arm ifuse NNthen d←G (arm) −G (1) c← q 2 logt N1 − q 2 logt Narm + (F(1) −F (arm)) else d←S (arm) −S (1) c← q 2 logt N1 − q 2 logt Narm −µ ⊤ 1 d Append(d, c)toC M ←∅ foreach(d, c)∈ Cdo M ← M ∪ {δ⊤d≥c+ 10 −6} δ←solve QP in...

    2026

  24. [1998]

    Neural tangent kernel: A survey

    Eugene Golikov, Eduard Pokonechnyy, and Vladimir Korviakov. Neural tangent kernel: A survey. arXiv preprint arXiv:2208.13614,

    Pith/arXiv arXiv

  25. [2014]

    Deep learning using rectified linear units (relu).arXiv preprint arXiv:1803.08375,

    Abien Fred Agarap. Deep learning using rectified linear units (relu).arXiv preprint arXiv:1803.08375,

    Pith/arXiv arXiv

  26. [2015]

    Genai-bench: Evaluating and improving compositional text-to-visual generation.arXiv preprint arXiv:2406.13743,

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation.arXiv preprint arXiv:2406.13743,

    Pith/arXiv arXiv

  27. [2016]

    algorithm is a simple bandit strategy that divides the learning process into two distinct phases. During the exploration phase, the algorithm 18 Accepted as a conference paper at ICLR 2026 Table 1: Comparison of attack performance on linear and non-linear reward models using theϵ- greedyandETCmethods. All attacks achieve a100%success rate (ASR= 100). The ...

    2026

  28. [2019]

    Multi-agent conversational online learning for adaptive llm response identification.arXiv preprint arXiv:2501.01849,

    Xiangxiang Dai, Yuejin Xie, Maoli Liu, Xuchuang Wang, Zhuohua Li, Huanyu Wang, and John Lui. Multi-agent conversational online learning for adaptive llm response identification.arXiv preprint arXiv:2501.01849,

    arXiv

  29. [2020]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851,

    11 Accepted as a conference paper at ICLR 2026 Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851,

    2026

  30. [2022]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  31. [2023]

    A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 1412(6),

    Kingma DP Ba J Adam et al. A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 1412(6),

    Pith/arXiv arXiv

  32. [2024]

    Teaching with limited information on the learner’s behaviour

    Ferdinando Cicalese, Eduardo Laber, Marco Molinaro, et al. Teaching with limited information on the learner’s behaviour. InInternational Conference on Machine Learning, pp. 2016–2026. PMLR,

    2016

  33. [2025]

    A multi-armed bandit approach to online selection and evaluation of generative models.arXiv preprint arXiv:2406.07451, 2024a

    Xiaoyan Hu, Ho-fung Leung, and Farzan Farnia. A multi-armed bandit approach to online selection and evaluation of generative models.arXiv preprint arXiv:2406.07451, 2024a. Xiaoyan Hu, Ho-fung Leung, and Farzan Farnia. An online learning approach to prompt-based selection of generative models and llms. InForty-second International Conference on Machine Lea...

    Pith/arXiv arXiv