{"id":"92398ceb-737b-4c7e-ae2b-b5d084d00171","arxiv_id":"2411.16591","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Two-window drift detectors can be silently evaded by constructing streams whose window averages stay equal while the underlying distribution changes.","lead":"This paper shows that common concept drift detectors can be fooled: an attacker can create data streams that genuinely change over time yet never trigger the drift alarm. It gives a mathematical characterization of exactly which changes go undetected and demonstrates the attacks in experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The finite-sample transfer of Theorem 1 is asserted rather than proven: Algorithm 2's kernel condition controls only window-averaged mixture weights, so the permutation-test p-value is unconstrained, and the one unexplained exception in Section 4 is the expected symptom.","rationale":"The reader's verdict is CONDITIONAL, and the identified weak point is exactly the finite-sample transfer. My stress-test confirms that this is the most load-bearing concern: the central practical claim of the paper rests on Algorithm 2 producing streams that evade the actual statistical test, but the only formal theorem is for the idealized limiting detector in Eq. (1). The proof of Theorem 1 is sound for that limiting case: the total-variation argument correctly reduces d(D_W1,D_W2)=0 to equality of all window-averaged probabilities, yielding the Adv0(A) characterization. The slip occurs when the paper moves to finite samples. The condition Wn v = 0 equalizes the expected mixture weights of the two windows, and therefore the population-level window distributions, but the permutation MMD p-value is computed from realized samples that are not exchangeable under the null hypothesis because each Xi follows its own mixture viP+(1−vi)Q. Hence the p-value is not guaranteed to be large, and the reported exception in Table 2 is direct evidence of this gap. I give credit for the empirical evaluation, which mostly aligns with the limiting theory, and for the reproducible code release; these support the paper's heuristic value. But the missing finite-sample guarantee is a genuine correctness risk, not merely a presentation issue. The reader's CONDITIONAL verdict already captures this; my proposed concrete test would determine whether the concern is a rare edge case or a systematic failure mode, and whether the paper needs a proof or an explicit heuristic framing.","tokens_in":9524,"tokens_out":13894,"duration_ms":132422,"concrete_test":"Reproduce Table 2 with an additional control: for a fixed window scheme (e.g., growing reference, a=150), generate v with Wn v = 0 in two regimes: (i) v chosen 'with as little changes as possible' as in the paper; (ii) v with the same window averages but maximal within-window oscillation (e.g., alternating 0/1 blocks that cancel across each window). Run the permutation MMD test (2500 permutations, theta=0.05) 10,000 times and compare empirical detection rates. Also rerun the exact exceptional 'Rand.Per. (150)/grow(150)' cell. If regime (ii) or the exceptional cell detects at a rate close to or above the nominal level, the finite-sample transfer fails; if both stay at or below theta, the heuristic is more robust than the proof suggests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest practical claim is that Algorithm 2 constructs finite streams that evade Algorithm 1. The load-bearing transfer is the assertion (Section 3.2, after Algorithm 2) that Wn v = 0 makes 'the mean distributions of the samples in W1 and W2 coincide ... which is what Algorithm 1 line 3 is testing for.' This is only true at the level of limiting window distributions. In the finite case, v_i varies within each window, so the samples Xi are not exchangeable, and the permutation MMD p-value is a random variable whose distribution is not controlled by Wn v = 0. The one exceptional cell reported in Section 4 is left unexplained; it is exactly the symptom one would expect from this gap. Consequently, the paper establishes a limiting-case characterization, but not a finite-sample guarantee that constructed streams pass unnoticed; the practical construction is an empirically supported heuristic whose failure mode is not characterized.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies adversarial attacks on concept-drift detectors that compare two windows of data. In an infinite-sample limiting model, it defines a detector A(D_t) = 1[sup_W d(D_{W1}, D_{W2}) > 0] and proves (Theorem 1) that undetected processes correspond exactly to mixing functions f in Adv0(A), the set of functions whose window averages coincide. It derives explicit descriptions of Adv0 for fixed, growing, and sliding reference windows (Table 1, Propositions 1–3) and proposes Algorithm 2 to construct finite-sample streams by solving Wn v = 0 and sampling Xi ~ v_i P + (1 − v_i) Q. Experiments on synthetic data and a water-network leak-detection showcase report mostly successful evasion, with one unexplained exception. The limiting-case theory is rigorous, but the finite-sample construction is asserted rather than proven to evade the permutation MMD test.","tokens_in":9702,"tokens_out":10620,"duration_ms":92638,"significance":"The limiting-case characterization is the paper's main strength: it is clean, fully proven in the appendix, and yields falsifiable predictions in the form of the function classes in Table 1. The paper also releases code, which supports reproducibility. If the finite-sample construction were proven, the paper would show a surprising and practically relevant vulnerability of common drift detectors. However, the practical claim rests on an unproven transfer from the limiting case to the permutation MMD test, and the one empirical exception is consistent with that gap. As submitted, the paper is a solid contribution to the theory of drift detection, while the finite-sample construction is an empirically supported heuristic whose failure mode is not characterized.","major_comments":[{"comment":"The assertion after Algorithm 2 that Wn v = 0 makes 'the mean distributions of the samples in W1 and W2 coincide ... which is what Algorithm 1 line 3 is testing for' is not justified. Wn v = 0 only forces the arithmetic means of the v_i over W1 and W2 to be equal, so the averaged mixture distributions (1/|W|) sum_{i in W} (v_i P + (1-v_i) Q) coincide. It does not imply that the individual points in W1 and W2 are drawn from the same distribution, and the permutation test's null hypothesis of exchangeability is not satisfied. Therefore the p-value is a random variable whose distribution is not controlled by Wn v = 0. The limiting-case Theorem 1 does not transfer automatically. Since the paper's practical claim to construct undetected finite streams relies on this step, a finite-sample guarantee (or an explicit statement that Algorithm 2 is a heuristic) is needed.","section":"Section 3.2 (Algorithm 2)"},{"comment":"The evaluation reports 'only one exceptional case' but does not identify or explain it. This exception matters because the empirical alignment is the main direct evidence for the finite-sample transfer. In particular, the Rand.Per. (150) cell with growing reference length 150 (0.02/0.00) is the kind of failure that the unproven finite-sample analysis would predict: Wn v = 0 does not control the permutation p-value, so occasional alarms are to be expected. The paper should either provide a proof that the p-value is stochastically large under the constructed distribution, or discuss and model the exception explicitly.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The decomposition in Proposition 2 is stated as 'f (x) = p(x) + tq(x)'; the variable t is free and should be x, i.e., f(x) = p(x) + x q(x) (or f(t) = p(t) + t q(t)). The proof uses the correct expression, so this is a typographical error, but it should be fixed.","section":"Proposition 2 (statement)"},{"comment":"The caption 'Correct result is p = 0, lining marks adversarials according to theory' is ambiguous: for adversarial streams the desired outcome is a large p-value (no alarm), while for non-adversarial streams it is a small p-value. Please rephrase to state what each cell is expected to show.","section":"Section 4, Table 2 caption"},{"comment":"The phrase 'lining marks adversarials according to theory' refers to visual formatting that is not visible in the text version. Either include the formatting or remove the reference to lining.","section":"Section 4, Table 2 caption"},{"comment":"The algorithm does not specify how to handle the case where the nullspace of Wn is trivial or where the normalized v is constant; in those cases no true adversarial stream is produced. The procedure should explicitly detect and report such a failure.","section":"Algorithm 2"},{"comment":"The window-averaged distributions D_{W1} and D_{W2} are used before being formally defined. A brief definition of D_W = (1/PT(W)) ∫_W D_t dPT(t) would improve readability.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious limiting-case study with full proofs, and the main theorem is sound. The central issue is the finite-sample transfer: the practical construction in Algorithm 2 is presented as if it inherits the guarantee of Theorem 1, but it does not, and the one unexplained empirical exception is a natural symptom of that gap. The authors could address this by either proving a finite-sample control (e.g., showing that the permutation MMD statistic is stochastically bounded under the constructed mixture) or by clearly reframing Algorithm 2 as a heuristic and toning down the abstract's claim that the paper shows how to construct undetected drifting streams. The paper is within the scope of the journal, with the caveat that much of the framing is theoretical; the water-network application is a brief showcase rather than a full evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know: this paper's limiting-case analysis of window-based drift detectors is correct and worth your time. Theorem 1 gives a clean necessary-and-sufficient condition for a two-window detector to miss drift: in the limit, drift is invisible iff for every measurable S the function t -> D_t(S) lies in Adv0(A), the set of functions whose window-averaged values coincide. Table 1 is the real payoff—explicit characterizations for fixed, sliding, and growing reference windows. Sliding windows admit exactly the l-periodic functions; fixed reference windows need l-periodicity after a with matching mean; growing reference windows force constancy after a (with boundary effects). The proofs are careful, and the observation that combining detectors shrinks Adv is a nice design rule.\n\nCredit where due: the paper makes precise what earlier work only constructed by example, and the empirical MMD evaluation mostly aligns with the theory, which is meaningful.\n\nSoft spots, in proportion. The finite-sample Algorithm 2 is the weak point. The assertion that Wnv = 0 makes 'the mean distributions of the samples in W1 and W2 coincide' is only true at the level of limiting window distributions. In finite samples, v can vary inside a window, so the permutation MMD p-value is not controlled; the 'one exceptional case' in Table 2 is exactly what that gap predicts. The paper should either prove a finite-sample guarantee or explicitly call Algorithm 2 an empirically supported heuristic. Second, Proposition 2's proof has a typo in the decomposition (q(t)+tq(t) should be p(t)+tq(t))—minor but confusing. Third, the water-network section is a showcase without quantitative results; it does not hurt the main claim.\n\nCitation pattern is fine; the authors cite their own survey and prior constructive example, which is relevant.\n\nWho is this for? Researchers working on drift detection robustness or monitoring security will get real value from Theorem 1 and Table 1. The finite-sample claims should not be taken as proven.\n\nRecommendation: send it to peer review. The core characterization deserves referee time; the gaps are fixable with a sharper statement of what Algorithm 2 actually guarantees and an explanation of the exceptional cell.","headline":"Limiting-case drift-adversarial characterization is solid and useful; the finite-sample construction has an unproven guarantee that the experiments only partially paper over.","tokens_in":10189,"tokens_out":3260,"would_cite":true,"duration_ms":30347,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves a necessary and sufficient condition for undetectable drift in two-window drift detectors: drift passes unnoticed exactly when the stream's time-indexed probabilities lie in the set of window-adversarial functions, and it…","keywords":["concept drift","drift detection","adversarial attacks","two-window detectors","distribution processes","maximum mean discrepancy","windowing schemes","water distribution networks"],"falsifier":"Construct a stream with $v\\in[0,1]^n$ satisfying $W_n v=0$ but oscillating much faster than the window length (e.g., alternating blocks of length 1), keep $P$ and $Q$ well separated, and run a standard permutation MMD two-window detector; if the p-value systematically falls below the threshold in repeated runs, the finite-sample transfer asserted by the paper fails. The paper's Table 2 already shows one unexplained exception of this kind.","tokens_in":9349,"feed_emoji":"⚠️","tokens_out":7945,"duration_ms":65843,"temperature":0.7,"pith_summary":"The paper establishes a precise mathematical condition under which a common class of concept-drift detectors — detectors that compare two sliding or reference windows — can be silently evaded. For the limiting no-false-positive detector, it proves that drift goes undetected exactly when the stream's probability of every measurable set, viewed as a function of time, lies in the set of 'window-adversarial' functions whose window averages coincide. Because these functions depend only on the windowing scheme and not on the distance metric, many standard detectors are vulnerable by construction. The paper gives explicit adversarial functions for fixed, sliding, and growing reference windows, a finite-sample construction algorithm, and empirical confirmation on synthetic and water-network data. If correct, the result turns drift detection into an auditable design problem: a detector can be certified to miss no drift only when its adversarial set is empty.","feed_headline":"Theorem pinpoints every drift a two-window detector can miss","feed_subtitle":"For fixed, sliding, and growing windows, genuine drift passes silently; the paper lists exactly when.","key_machinery":"The paper's central machinery is the family of window-average operators encoded in a weight matrix $W_n$: each window pair $(W_1,W_2)$ is represented by the vector $w=|W_1|^{-1}\\sum_{i\\in W_1}e_i - |W_2|^{-1}\\sum_{i\\in W_2}e_i$, so an adversarial drift pattern is a vector $v\\in[0,1]^n$ in the kernel of $W_n$. In the infinite-sampling limit the same idea becomes the function space $\\mathrm{Adv}_0(A)$ of time-indexed mixing weights whose averages over every compared pair of windows are equal. The load-bearing identity in Theorem 1 is the equivalence between equality of window distributions, equality of the corresponding window averages of $t\\mapsto D_t(S)$, and membership in $\\mathrm{Adv}_0(A)$. This converts the question 'can this detector be fooled?' into a linear-algebra computation, which the paper solves in closed form for fixed, sliding, and growing reference windows.","core_discovery":"On the paper's own terms, the central discovery is a characterization of undetectable drift for two-window detectors. In the limiting case the detector $$A(D_t)=\\mathbf{1}\\left[\\sup_{W} d(D_{W_1},D_{W_2})>0\\right]$$ has no false positives, so its only failures are false negatives; Theorem 1 states that $A$ fails to detect drift precisely when, for every measurable $S$, the time-indexed probability function $t\\mapsto D_t(S)$ belongs to $\\mathrm{Adv}_0(A)$, the set of functions satisfying $P_T(W_2)\\int_{W_1} f\\,dP_T = P_T(W_1)\\int_{W_2} f\\,dP_T$ for every compared window pair. Any non-constant $f$ in that set yields a genuinely drifting stream $D_t=f(t)P+(1-f(t))Q$ that triggers no alarm, so an empty adversarial set $\\mathrm{Adv}(A)$ is necessary and sufficient for the detector to detect every drift when $d$ is a metric. A second key point is that this vulnerability depends only on which data windows are compared, not on the statistical distance used, and that combining detectors shrinks the adversarial set to the intersection of the individual sets.","pith_inferences":["A natural extension the paper does not pursue is the role of oscillation speed: very fast oscillations of $f$ approach noise and may be suppressed by chunked sampling, while very slow oscillations approach the constant (non-drifting) case, so the practically evasive frequency band is an empirical question left open.","The characterization provides an auditing recipe for monitoring systems: compute the adversarial set of a proposed detector and check whether naturally expected drift patterns (daily or weekly cycles, sensor aging) fall inside it; the water-network showcase already hints that window length can be chosen so that harmless periodic patterns become the adversarial ones.","The same window-average mechanism should apply to any detector that compares the mean distribution of two windows, such as per-pair Kolmogorov–Smirnov or energy-distance tests, even though the paper only constructs explicit adversarials for the MMD-based setting."],"forward_implications":["Any two-window detector whose reference scheme admits a non-constant adversarial function can be evaded by a stream that mixes two fixed distributions with that time-varying weight, no matter what distance metric it uses.","A detector provably catches all drifts if and only if its adversarial set is empty; for the three common reference-window strategies the paper computes these sets explicitly, exposing periodic and boundary-effect blind spots.","Combining detectors, for instance alarming whenever any one of several window schemes fires, reduces the undetectable class to the intersection of the individual adversarial sets, making composition a concrete defence.","The finite-sample construction (Algorithm 2) turns the characterization into a practical attack: choose $v$ in the kernel of the window-encoding matrix, sample each observation from $v_iP+(1-v_i)Q$, and the compared windows have equal mean distributions."],"supporting_citations":[{"why":"Supplies the distribution-process model and the catalogue of windowing schemes (fixed, growing, sliding) that the adversarial functions are computed for.","marker":"[1]"},{"why":"Gives the sample-wise definition of concept drift that the finite-case construction targets.","marker":"[3]"},{"why":"Provides a constructive proof that can be modified to build metric adversarials, the starting point for the paper's discussion of metric-based detectors.","marker":"[7]"},{"why":"Supplies the water-distribution-network application and the expected daily and weekly pressure patterns used in the showcase.","marker":"[4]"},{"why":"Provides the Shape Drift Detector postprocessing used in the water-network evaluation to turn raw MMD values into candidate drift points.","marker":"[8]"}],"fun_headline_variants":["Two-window detectors can miss drift — here's the exact loophole","Exact conditions under which drift slips past two-window tests","Drift can hide from any two-window detector — here's when","Characterizing all undetectable drifts for two-window detectors","Invisible drift: the exact set of streams that fool two-window detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis treats a drift detector as a deterministic device that compares the exact distributions of two windows and raises an alarm whenever those distributions differ; the paper's practical algorithm assumes a finite statistical test (such as a permutation MMD test) honors that same comparison, an assumption asserted but not proven.","fun_headline_variants_meta":{"raw":{"variants":["Two-window detectors can miss drift — here's the exact loophole","Exact conditions under which drift slips past two-window tests","Drift can hide from any two-window detector — here's when","Characterizing all undetectable drifts for two-window detectors","Invisible drift: the exact set of streams that fool two-window detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000902,"raw_usage":{"total_tokens":3859,"prompt_tokens":900,"completion_tokens":2959,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2865}},"tokens_in":516,"tokens_out":2959,"duration_ms":21018,"temperature":1.0,"reasoning_tokens":2865,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:57:05.587534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a stream with $v\\in[0,1]^n$ satisfying $W_n v=0$ but oscillating much faster than the window length (e.g., alternating blocks of length 1), keep $P$ and $Q$ well separated, and run a standard permutation MMD two-window detector; if the p-value systematically falls below the threshold in repeated runs, the finite-sample transfer asserted by the paper fails. The paper's Table 2 already shows one unexplained exception of this kind.","supporting_citations":[{"cited_title":"Hinder, V","cited_arxiv_id":null,"evidence_quote":"Supplies the distribution-process model and the catalogue of windowing schemes (fixed, growing, sliding) that the adversarial functions are computed for."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the sample-wise definition of concept drift that the finite-case construction targets."},{"cited_title":"Hinder, V","cited_arxiv_id":null,"evidence_quote":"Provides a constructive proof that can be modified to build metric adversarials, the starting point for the paper's discussion of metric-based detectors."},{"cited_title":"Vaquet, F","cited_arxiv_id":null,"evidence_quote":"Supplies the water-distribution-network application and the expected daily and weekly pressure patterns used in the showcase."},{"cited_title":"A detects drift if and only if Dt has drift","cited_arxiv_id":null,"evidence_quote":"Provides the Shape Drift Detector postprocessing used in the water-network evaluation to turn raw MMD values into candidate drift points."}],"review_version":1}