{"id":"ae2b4218-8f7d-41f9-9b58-0082fbbcc192","arxiv_id":"2412.08099","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A simple gradient-free 'directional gradient approximation' attack makes LLM time series forecasters degrade more than equivalent random noise, across GPT-3.5, GPT-4, LLaMa, Mistral, TimeGPT, and TimeLLM.","lead":"This paper shows that small, deliberately chosen changes to a time series can make LLM-based forecasters produce badly wrong predictions, much worse than random noise of the same size. It tests a simple black-box attack, DGA, across five datasets and several models to expose how fragile these forecasters are.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 5's 'gradient' is a scalar loss difference divided elementwise by a random vector, so the DGA perturbation sign is essentially random; the paper never specifies theta, query count, or why this is an ascent direction.","rationale":"The reader's weakest assumption and my load-bearing concern coincide: the DGA gradient estimator in Eq. 5 is under-specified and not derived, and without it the attack's advantage over random noise is unexplained. I agree with the reader's conditional verdict because the empirical direction is plausible and the released code could resolve the ambiguity, but the paper as written does not yet support the strong claims about targeted adversarial effectiveness or LLM-specific fragility. My concrete test would settle the core question: if DGA is equivalent to a random sign perturbation with a global flip, then the central mechanism is not a gradient approximation, and the headline comparisons to GWN would need to be reinterpreted. I did not identify a separate fatal flaw; the main issue is that the mathematical specification of the attack is internally inconsistent as written. The reader's requested additions of error bars, proper baselines, and matched training are all reasonable, but the Eq. 5 issue is the most load-bearing because it underpins the attack's identity. If the code check shows DGA matches random sign, the verdict should move toward rejection or unverified; if it shows a coordinate-wise or multi-probe estimator with a clear query budget, the conditional acceptance can stand with clarifications.","tokens_in":11643,"tokens_out":7020,"duration_ms":83677,"concrete_test":"Run the released code on one dataset/model pair, instrumenting the number of queries and the sign pattern of rho. Fix a symmetric theta (e.g., uniform(-eta, eta)^T), compute Eq. 5 exactly, and record rho. Then compare three attacks at identical perturbation budgets and identical query counts over at least 10 random seeds: (1) DGA as implemented, (2) a random sign attack with rho_i = +/- epsilon and no queries, and (3) a coordinate-wise finite-difference sign gradient using theta = eta * e_i. If rho from Eq. 5 equals epsilon * sign(theta) up to a global sign, or if DGA is not statistically better than the random sign attack, then Eq. 5 contributes no gradient information and the reported DGA-versus-GWN advantage is not evidence for a targeted attack.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on DGA being a genuinely gradient-informed, targeted attack. As written, Eq. 5 defines g_{t-T+1:t} = [L(Y - f(X + theta)) - L(Y - f(X))] / theta, where theta is called a random small signal. The numerator is a scalar, and the division is elementwise. Consequently, sign(g_i) = sign(delta) * sign(theta_i), where delta is the scalar loss difference. The perturbation in Eq. 6 is thus rho_i = epsilon * sign(theta_i) up to a single global sign chosen by whether one random probe increased or decreased the loss. This is a one-bit line search along a random direction, not a gradient approximation. The paper does not specify the distribution of theta, the number of queries, or how multiple probes are combined. If theta is a scalar, sign(g) is scalar and the perturbation is constant across time, which is inconsistent with the reported time-varying perturbations; if theta is a vector, the per-coordinate sign is just the sign of the random probe. In neither case does Eq. 5 establish an ascent direction. If the actual implementation loops over coordinates with theta = eta * e_i, that loop is hidden and the query cost is T per example, which must be stated. The comparison to Gaussian white noise is also ambiguous: 'same perturbation intensity' is undefined, and if GWN is not clipped to the epsilon ball while DGA is, DGA has a larger effective norm. The headline claims of minimal yet highly effective perturbations and the LLM-versus-non-LLM robustness comparison rest on this unvalidated estimator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a black-box adversarial attack, Directional Gradient Approximation (DGA), against LLM-based time series forecasters. The attack approximates a gradient by a finite difference along a random signal, applies an epsilon-bounded sign perturbation, and targets Gaussian white noise (GWN) whose mean and standard deviation are taken from the input window. Experiments cover five datasets (ETTh1, ETTh2, IstanbulTraffic, Weather, Exchange) and eight models (LLMTime with GPT-3.5/GPT-4/LLaMa/Mistral, TimeLLM, TimeGPT, TimesNet, iTransformer), reporting MSE/MAE under clean input, GWN, and DGA. The authors conclude that DGA degrades forecasts substantially more than GWN and that LLM-based forecasters are less resilient than non-LLM models.","tokens_in":11920,"tokens_out":7015,"duration_ms":67306,"significance":"If the claims were fully supported, the paper would provide the first broad demonstration that API-accessible LLM forecasters can be strongly perturbed without gradient access, which is relevant to the reliability of LLM-based forecasting in finance, energy, and traffic. The experimental breadth (five datasets, eight models, three LLM application paradigms) is a genuine strength, and the direct comparison of DGA against GWN is a meaningful non-circular control for perturbation strength. The public code repository is also a positive feature. However, as detailed below, the gradient estimator in Section 4 is not derived, the empirical tables lack error bars and significance tests, the noise baseline is under-specified, and part of the 'noise-like output' interpretation follows directly from the choice of target. Thus the contribution is significant only conditionally on a substantial revision of the method and evidence.","major_comments":[{"comment":"The proposed 'gradient' estimator is not a valid ascent (or descent) direction as written. The numerator L(Y - f(X+theta)) - L(Y - f(X)) is a scalar, so dividing by the random vector theta elementwise gives g_i = delta / theta_i, and Eq. (6) implies sign(rho_i) = sign(delta) * sign(theta_i). Up to one global sign chosen by the single probe, the perturbation is the sign pattern of the random vector theta and contains no coordinatewise gradient information. The manuscript does not specify the distribution of theta, the number of probes, or how multiple probes are combined; if theta is a scalar the perturbation is constant across time, while if theta is a vector the signs are random. To support the claim that DGA is gradient-informed, the authors must either derive a proper coordinate-wise estimator with its bias/query cost or explicitly state and justify the actual algorithm used (e.g., a per-coordinate finite-difference loop with theta = eta*e_i).","section":"Section 4, Eq. (5)"},{"comment":"The central empirical claim that DGA is 'much more detrimental' than GWN is not supported by the reported aggregate numbers, which have no error bars, no repeated trials, and no significance tests. Several entries show differences in the opposite direction or within rounding noise: TimeLLM on ETTh1 has MSE 0.099 under DGA versus 0.102 under GWN; TimeGPT on ETTh1 has 0.060 vs 0.059; iTransformer on ETTh2 has 0.179 vs 0.181; TimeNet on ETTh2 has 0.169 vs 0.166. Because the comparison is the paper's main evidence for attack effectiveness and for the LLM-versus-non-LLM robustness conclusion (Section 5.4, Figure 2), the tables should report means over multiple seeds with standard deviations and a paired significance test or effect-size measure.","section":"Table 1"},{"comment":"The phrase 'same perturbation intensity' is undefined for the GWN baseline. The caption states that the perturbation scale is 2% of the mean value of each dataset, but it does not state the standard deviation or distribution of the GWN, whether it is generated independently per coordinate, or whether it is clipped to the same l_infinity ball as DGA's epsilon constraint in Eq. (6). If DGA is always at norm epsilon while GWN has a smaller effective norm, part of DGA's larger error increase is an artifact of larger perturbation energy. Please define the exact GWN construction and report the achieved perturbation norms for both methods.","section":"Section 5.3, Table 1 caption"},{"comment":"The interpretation that the attack makes forecasts 'random-walk-like' or 'noise-like' is partly built into the objective. In Section 4 the target sequence Y in Eq. (4) is explicitly chosen as GWN with statistics taken from the input, so the optimization is designed to drive f(X+rho) toward GWN; the distribution plots and ACF analysis in Figures 4 and 5 therefore show that the objective was optimized, not that a novel vulnerability of LLMs was discovered. The DGA-versus-GWN error comparison is a fair control and should remain, but the claims in Section 5.5 should be tempered or supplemented with experiments against alternative target sequences (e.g., shifted, constant, or periodic targets) to test whether the effect is specific to noise targets.","section":"Section 4 and Section 5.5"},{"comment":"The threat model is black-box with query access, but the paper never reports the query count or cost of DGA. If the estimator in Eq. (5) requires one query per random probe and the implementation actually loops over coordinates, the total query budget per example is at least T (or more), which must be stated to evaluate whether the attack is 'minimal' in any operational sense. Reporting query counts is also necessary to assess the practical threat against API-based models such as GPT-3.5 and GPT-4.","section":"Section 4 and Section 5.3"}],"minor_comments":[{"comment":"The code link is incomplete: 'Johnson/AdvAttackLLM4TS' should be a full URL or repository path.","section":"Abstract and Section 5"},{"comment":"The header contains a typo ('Metrcis') and the column label 'iTransformerTimesNet' should be split into two separate model columns to avoid ambiguity.","section":"Table 1"},{"comment":"The loss notation is inconsistent: Eq. (2) writes L(f(...), Y) with two arguments, while Eq. (5) writes L(Y - f(...)); please unify the notation and define the arguments of L.","section":"Section 4"},{"comment":"The procedure 'we introduced GWN with the same perturbation intensity' does not specify how the GWN target is generated per test window or whether mu and sigma are computed from the clean input window; please clarify.","section":"Section 5.3"},{"comment":"The radar charts plot 'normalized increase in MAE', but the normalization base (clean MAE per dataset, per model, or global) is not defined; please define it in the caption or text.","section":"Figure 2"},{"comment":"The statement that TimeLLM is fine-tuned on 10% of each dataset should clarify whether this 10% is part of the training split and whether the attacker has access to any fine-tuning data, given the threat model's 'no access to training data' assumption.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the experimental scope is a strength. However, the main algorithm as presented in Eq. (5) does not define a gradient estimator, and the empirical support is weakened by the lack of error bars and an under-specified GWN baseline. I do not see a fundamental novelty problem; the issues are technical and, in principle, fixable with a rewritten method section and additional experiments. I also note that the code link is incomplete and should be checked before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to the chase. This is the first black-box attack study, within its own cited literature, aimed specifically at LLM-based time series forecasters, and the breadth is real: LLMTime with four base LLMs, TimeGPT, TimeLLM, five datasets, plus two non-LLM comparators. The threat model is honest—no ground truth, no gradients, no training data, query-only—and the qualitative conclusion that these models can be pushed toward noise-like predictions by small input perturbations is plausible. That alone makes the paper worth engaging with.\n\nThe soft spots are concentrated in the method and the statistics, and they are not minor. Equation 5 does not define a gradient estimator as written. The numerator is a scalar loss difference; dividing elementwise by a random vector yields sign(g_i) = sign(delta) * sign(theta_i), so the perturbation is essentially a one-bit random-direction step with a global sign from a single probe. If the implementation actually loops over coordinates with theta = eta * e_i, the paper needs to say so and report the query cost. This step is load-bearing because the whole attack is built on it. Table 1 also has no error bars, repetitions, or significance tests; many of the reported DGA-versus-GWN differences are small (e.g., TimeGPT on ETTh1: 0.059 to 0.060), so \"significantly degrade\" is not established by the numbers as shown. The only baseline is Gaussian noise; an existing black-box attack or even random sign perturbation would tell you how much of the effect is specific to the directional heuristic. And the non-LLM comparison is confounded: iTransformer and TimeNet get full training data while the LLMs are zero-shot or 10% fine-tuned, so \"less resilient\" is partly a training-data effect.\n\nSome things are less damning than they look. The DGA-versus-GWN comparison is a genuine control against the circularity of targeting Gaussian statistics, and the hyperparameter study at different scales is useful. The code is promised, which at least makes the ambiguity in Eq. 5 checkable.\n\nWho is this for? People working on robustness and deployment of LLM forecasters, and adversarial time series. It deserves a serious referee, but only with a request for major revision: clarify or fix the estimator, add repeated trials with confidence intervals, add a stronger baseline, and match training setups. I would not cite it in its current form.","headline":"First black-box attack on LLM forecasters with broad coverage, but Eq. 5's gradient is not what it claims and the evidence lacks error bars; worth a serious referee, not acceptance as is.","tokens_in":12508,"tokens_out":3287,"would_cite":false,"duration_ms":35467,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that query-only adversarial perturbations, built from a crude finite-difference gradient estimate, push LLM time-series forecasters toward noise-like predictions far more effectively than random perturbations of the same…","keywords":["adversarial attacks","time series forecasting","large language models","black-box attack","gradient-free optimization","directional gradient approximation","robustness","Gaussian white noise target"],"falsifier":"Run DGA versus random-sign perturbations with the same epsilon budget across many seeds on one of the reported models and datasets; if the two produce statistically indistinguishable MSE or MAE increases, the directional estimate carries no information and the central claim fails. Alternatively, vary the distribution and magnitude of theta in Eq. 5; if DGA's advantage disappears under reasonable choices, the proposed gradient proxy is not stable enough to support the headline effect sizes.","tokens_in":11386,"feed_emoji":"📉","tokens_out":5445,"duration_ms":52101,"temperature":0.7,"pith_summary":"This paper tries to establish that LLM-based time series forecasting models are practically attackable by an adversary who can only query the model. It introduces Directional Gradient Approximation (DGA), a gradient-free black-box attack that requires no access to model weights, training data, or future ground truth. With perturbations set to just 2% of each dataset's mean, DGA degrades forecasting accuracy far more than Gaussian white noise of the same intensity across five datasets and six LLM-based model variants. The paper also claims that LLM forecasters are less resilient than two non-LLM transformer models under the same attack. If true, this means real-world deployments of LLM forecasters in finance, energy, and traffic systems need robustness defenses before they can be trusted.","feed_headline":"Tiny perturbations make LLM forecasters predict noise","feed_subtitle":"A query-only attack degrades LLM forecasters far more than random noise, and simpler models prove sturdier.","key_machinery":"The load-bearing object is the Directional Gradient Approximation (DGA), defined by the quotient $$g_{t-T+1:t} = \\frac{L(Y - f(X_{t-T+1:t} + \\theta_{t-T+1:t})) - L(Y - f(X_{t-T+1:t}))}{\\theta_{t-T+1:t}},$$ with $\\theta$ a random small signal, $Y$ the targeted anomalous output (chosen as Gaussian white noise), and $L$ the loss between the model output and the target. The perturbation is then $\\rho = \\epsilon \\cdot \\operatorname{sign}(g)$, applied under an $\\ell_1$ budget. This quotient is treated as a usable ascent direction despite being a single finite-difference sample with no stated distribution or bias analysis; it is what lets the attack work without gradients or internal access.","core_discovery":"The central claim is that a black-box attacker with only query access can craft minimal perturbations that substantially degrade the forecasts of LLM-based time series models. The attack, DGA, replaces the unavailable gradient by a directional finite-difference quotient along a random small signal, flips its sign, and applies it as an epsilon-bounded perturbation (Eq. 5 and Eq. 6). Using Gaussian white noise as the target sequence, the attack shifts model outputs toward i.i.d. noise and destroys temporal correlation in the forecasts. Across ETTh1, ETTh2, IstanbulTraffic, Weather, and Exchange datasets, DGA produces larger MSE and MAE increases than Gaussian white noise of equal intensity, and LLM-based models show larger error increases than the two non-LLM transformer baselines, supporting the paper's conclusion that LLM forecasters are more vulnerable.","pith_inferences":["Because DGA uses only input-output queries and a target distribution, the same recipe could be tested against non-LLM forecasters and against multivariate or longer-horizon settings not covered in the paper; a wider baseline sweep would show whether 'less resilient' is specific to LLMs or common to sequence models.","The choice of Gaussian white noise as the target suggests a statistical defense: monitoring forecast autocorrelation or distributional distance from the input could flag DGA-style attacks without retraining the model.","Generalizing the target from white noise to a chosen misleading sequence (for example, a flat or inverted trend) would turn the attack from destroying accuracy into injecting a specific wrong forecast, which is the higher-stakes threat for real applications.","The reported effect sizes use a perturbation scale of 2% of the dataset mean, so whether the attack is truly 'imperceptible' depends on dataset variance and downstream tolerance; the practical threat level should be re-examined per deployment rather than taken as universal."],"forward_implications":["A practical attacker needs only query access: no weights, no training data, and no future ground truth are required to build effective perturbations.","Small perturbations at 2% of the dataset mean can make temporally structured forecasts look like independent noise, so forecast integrity can collapse under attack.","LLM-based forecasters (TimeGPT, LLMTime variants, and TimeLLM) are less resilient than the tested non-LLM transformer models, so adversarial robustness should factor into model selection.","Adversarial training is computationally prohibitive for large pretrained forecasters, so preprocessing filters and anomaly detection are the paper's suggested practical defenses.","The findings motivate robustness auditing before deploying LLM forecasters in high-stakes domains such as finance, energy demand prediction, and intelligent transportation."],"supporting_citations":[{"why":"Supplies LLMTime as the zero-shot LLM forecaster and the IstanbulTraffic dataset used in the experiments.","marker":"Gruver et al., 2024"},{"why":"Supplies TimeLLM, the fine-tuned LLM forecaster targeted by the attack.","marker":"Jin et al., 2023a"},{"why":"Supplies TimeGPT, the time-series-pretrained LLM forecaster targeted by the attack.","marker":"Garza and Mergenthaler-Canseco, 2023"},{"why":"Supplies TimeNet as a non-LLM baseline and the Weather dataset.","marker":"Wu et al., 2023"},{"why":"Supplies iTransformer as the second non-LLM baseline for the robustness comparison.","marker":"Liu et al., 2024d"},{"why":"Establishes that forecasting attacks cannot use ground truth values, which motivates the paper's label-free threat model.","marker":"Liu et al., 2022a"},{"why":"Provides the sign-based perturbation idea that Eq. 6 adapts for the black-box setting.","marker":"Goodfellow et al., 2014"},{"why":"Supplies the ETTh1 and ETTh2 electricity transformer temperature datasets used in the evaluation.","marker":"Zhou et al., 2021"},{"why":"Supplies the Exchange dataset of daily exchange rates used in the evaluation.","marker":"Lai et al., 2018"}],"fun_headline_variants":["Black-box attack flips LLM forecasts toward noise","LLM forecasters more fragile than simple transformers","Query-only attack fools LLM forecasters with tiny edits","LLM time-series models crack under adversarial push"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the one-shot finite-difference quotient in Eq. 5, taken along a random small signal and sign-flipped, is a reliable ascent direction; the paper gives no derivation or bias analysis for it, so the attack's advantage over random noise rests on that estimator working.","fun_headline_variants_meta":{"raw":{"variants":["Black-box attack flips LLM forecasts toward noise","LLM forecasters more fragile than simple transformers","Query-only attack fools LLM forecasters with tiny edits","LLM time-series models crack under adversarial push"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1514,"prompt_tokens":907,"completion_tokens":607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":523,"tokens_out":607,"duration_ms":6933,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:13:41.927397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DGA versus random-sign perturbations with the same epsilon budget across many seeds on one of the reported models and datasets; if the two produce statistically indistinguishable MSE or MAE increases, the directional estimate carries no information and the central claim fails. Alternatively, vary the distribution and magnitude of theta in Eq. 5; if DGA's advantage disappears under reasonable choices, the proposed gradient proxy is not stable enough to support the headline effect sizes.","supporting_citations":[{"cited_title":"Modeling long-and short-term temporal patterns with deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the Exchange dataset of daily exchange rates used in the evaluation."}],"review_version":1}