{"id":"17aa583b-1f77-4053-9837-56bd4ad9414d","arxiv_id":"2501.04130","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Any finite-ARL multi-stream change detector has trivial worst-case FDR, FWER, PFER, and GER; this paper defines error-over-patience (EOP) and gives e-detector based procedures that control EOP uniformly over stopping times.","lead":"This paper proves that any change-detection algorithm with a finite average run length cannot also control standard false-detection error rates, and introduces a new metric, error over patience, together with e-value based algorithms that provide non-trivial guarantees. The work is relevant to any system that monitors many data streams and must balance false alarms against detection speed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main EOP theorems hinge on each e-detector being an e-detector w.r.t. the global filtration; Section 8.2 shows this can fail under simple cross-stream dependence, so the 'very general dependence' claim is overstated.","rationale":"The reader's weakest assumption and the present stress-test converge: the global-filtration e-detector requirement is the point where the central EOP control claim is least secure. The paper's own Section 8.2 supplies a minimal example where the condition fails, and Section 8.1 admits that dependence is only permitted if it 'remains in the past.' This does not refute any internal theorem—the theorems explicitly state the condition—but it does mean the abstract's 'very general dependence structures' is an overstatement. In particular, the applicability of the proposed methods to dependent multi-stream data is narrower than the abstract suggests. The impossibility results and the proofs of Theorems 4.2/5.3/7.2 are otherwise sound; the e-detector stopping-time manipulation is valid because the assumption quantifies over all global null distributions. I do not find a more serious internal inconsistency. The Theorem 4.4 proof issue noted by the reader is real but peripheral: the BH-based PFA argument is not justified as written, yet the PFA bound follows directly from the uniform FDR bound by applying it at the first-false-alarm stopping time. Thus the verdict should remain conditional: the main EOP theorems are correct under a clearly stated assumption, but the scope of the dependence claim and a secondary proof gap need revision.","tokens_in":32799,"tokens_out":17915,"duration_ms":173135,"concrete_test":"Simulate the Section 8.2 counterexample: X_t^(1) ~ N(0,1) iid, X_t^(2)=X_{t+1}^(1), K=2, global null. Construct the SR e-detector for stream 1 from X_1^(1),... (and similarly for stream 2) and run e-d-BH with α=0.05. Evaluate the EOPFDR over a family of F-stopping times that use X_t^(2), e.g., τ_c = inf{t : X_t^(2) > c} and mixtures of such times; compute sup over the family of FDR(τ)/E[τ]. If any ratio exceeds 0.05, the EOP control fails under this dependence, confirming that the global-filtration condition is load-bearing and not a mere proof artifact. An analytic version: show directly that for c chosen so P(X_1^(2)>c) is non-negligible, E[M^(1)_{τ_c}] > E[τ_c], violating Definition 2.12 for the global filtration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central EOP guarantees (Theorems 4.2, 5.3, and 7.2) all assume that each M^(k) is an e-detector with respect to the global filtration F, i.e., E_P[M^(k)_τ] ≤ E_P[τ] for every F-stopping time τ and every global null distribution for stream k. This is strictly stronger than the per-stream e-detector property. Section 8.2 gives a concrete dependence structure, X_t^(2)=X_{t+1}^(1), under which a per-stream e-detector for stream 1 is not an e-detector for the global filtration: the global σ-field at time t contains X_{t+1}^(1), so an F-stopping time can exploit future information and violate the inequality. Since this inequality is exactly where the EOP bounds bite, the theorems do not cover this (and many related) cross-stream dependence structures. The abstract says the algorithms control EOP 'under very general dependence structures,' but Section 8.2 demonstrates that the condition is restrictive: essentially, no stream may see the future of another, and per-stream e-detectors must remain valid when optional stopping is taken w.r.t. the enriched global filtration. The positive examples in Section 8.3 are carefully constructed to satisfy this condition. The mathematical results are internally correct, and the paper is transparent in Section 8, but the headline claim in the abstract overstates the breadth of the dependence assumptions; the EOP guarantees are really conditional on a strong global-filtration e-detector assumption that is not preserved under arbitrary cross-stream dependence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies multiple-testing error control in multi-stream sequential change detection. The authors prove that any monitoring algorithm with finite average run length (ARL) has trivial worst-case FDR, FWER, PFER, and GER over all stopping times, motivating a new class of metrics called error-over-patience (EOP). They introduce four procedures—e-d-BH, e-d-Bonferroni, e-d-Holm, and e-d-GNT—that combine e-detectors with e-value multiple-testing methods, and prove EOP control as well as universal Type I error control when ARL finiteness is forfeited. The paper also discusses dependence assumptions, gives positive and negative examples, and reports simulations for parametric, nonparametric, and conformal change-detection settings.","tokens_in":33105,"tokens_out":19227,"duration_ms":189011,"significance":"The paper addresses a real gap: most multi-stream change-detection theory controls ARL or detection delay but not false-detection metrics across streams, while the multiple-testing literature does not treat sequential stopping. The impossibility results (Theorems 4.1, 5.1, Proposition 5.2, Proposition 7.1) are simple but important, and the EOP framework is a sensible response to the impossibility. The proposed algorithms are broadly applicable, come with explicit guarantees uniformly over stopping times, and are supported by reproducible simulations with provided code. The paper is transparent about the global-filtration assumption needed for EOP control and provides both positive and negative dependence examples. However, several proof details in the universal-control sections are incorrect as written, and the abstract overstates the dependence generality, so the paper needs revision before it can be accepted.","major_comments":[{"comment":"The proof of Theorem 4.4 is invalid in its current form. It argues that the event of any false detection is contained in the event that the BH procedure rejects a null when applied to p-values 1/(sup_t M_t). First, the standard BH procedure does not control FDR under arbitrary dependence among the p-values; it requires a condition such as PRDS, which is not established here. The correct tool is the e-BH theorem of Wang and Ramdas (2022), which controls FDR for arbitrarily dependent e-values. Second, replacing M_T by sup_t M_t does not preserve the e-variable property: an e-process need not have E[sup_t M_t] <= 1, so sup_t M_t is not guaranteed to be an e-variable. The theorem statement is likely true, but a correct proof should use the fact that the average (1/K) sum_k M_t is an e-process and apply Ville's inequality to bound the probability of any rejection across all t; the current proof needs to be rewritten.","section":"Theorem 4.4"},{"comment":"The proof of Corollary 4.2.1 asserts that FDR(phi, xi, tau*_1(phi, xi)) = 1 for arbitrary configurations xi. This is false when true and false detections occur at the same time: at the first false detection, the false discovery proportion can be strictly less than 1 if true detections are also present. A correct proof can be obtained from the inequality 1_{tau<infinity} <= sum_{k in H0} phi^{(k)}_tau and the e-detector property, which gives E[tau] >= 1/alpha; the current proof should be replaced. Similarly, the proof of Corollary 5.3.1 cites Theorem 4.2 instead of Theorem 5.3 and states PFER >= eta without conditioning; the statement itself is true when the ARL is finite, but the proof needs to be written correctly.","section":"Corollary 4.2.1 and Corollary 5.3.1"},{"comment":"The abstract and introduction claim that the EOP guarantees hold 'under very general dependence structures' and 'under rather general settings and weak assumptions.' This overstates the actual condition: Theorems 4.2, 5.3, and 7.2 require each M^{(k)} to be an e-detector with respect to the global filtration F, not just the per-stream filtration. Section 8.2 gives a concrete example, X_t^{(2)} = X_{t+1}^{(1)}, where a per-stream e-detector is not a global-filtration e-detector, so the EOP theorems do not apply to that dependence structure. The paper is commendably transparent in Section 8, but the abstract and introduction should be qualified to say that EOP control holds under dependence structures that preserve the e-detector property under the global filtration (e.g., independent streams or the examples in Section 8.3), rather than under 'very general dependence.'","section":"Abstract and Section 8"}],"minor_comments":[{"comment":"The condition in Algorithm 2 uses K/alpha_t, but the input parameter is (beta_t); it should be K/beta_t to match Theorem 5.3 and the proof.","section":"Algorithm 2, line 7"},{"comment":"The proof cites 'Theorem 4.2' when it should cite Theorem 5.3, and it writes ARL1(phi*, xi) with a star that is inconsistent with the notation elsewhere.","section":"Corollary 5.3.1 proof"},{"comment":"The proof refers to 'step T' without defining T clearly; it appears to mean an arbitrary stopping time, but the notation should be made consistent with the theorem's tau.","section":"Theorem 4.4 proof, notation"},{"comment":"The heading 'Controlling the worst-case PFER with a bounded PF A' should likely read 'PF_A' or 'probability of false alarm' to avoid confusion with PFER.","section":"Section 5.6 heading"}],"recommendation":"major_revision","confidential_remarks":"The paper is mathematically interesting and the main EOP theorems have clean, correct proofs. The issues are localized: the proof of Theorem 4.4 is wrong in its current form, and two corollaries have incorrect proof steps, though the statements appear salvageable. The dependence overstatement in the abstract is fixable by rewording. With these corrections, the paper would be a solid contribution. I see no citation or novelty concerns beyond normal expectations for a math.ST paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jin,\n\nRead the Dandapanthula–Ramdas paper. The core is stronger than the packaging. The impossibility result (finite ARL forces FDR/FWER/PFER/GER to degenerate) is elementary but worth stating cleanly, and the EOP metric is a sensible way out: control Type I error per unit of expected stopping time, uniformly over stopping times and configurations. The e-d-BH, e-d-Bonferroni, and e-d-GNT procedures with uniform stopping-time EOP bounds are a genuine extension of e-BH/e-Bonferroni to the sequential multi-stream setting. I checked the main proofs (4.2, 5.3, 7.2) and they go through: the bounds are direct consequences of the e-detector inequality E[M_tau]≤E[tau]. The universal Type I error control theorems (4.3, 5.4, 6.1, 7.3) also look correct. They are transparent in Section 8 about the global-filtration condition.\n\nSoft spots, in order:\n\n- Theorem 4.4's proof is wrong as written. It converts e-processes to p-values 1/sup_t M_t and appeals to BH for FDR control. BH does not control FDR under arbitrary dependence, and these p-values are not independent. The theorem itself is true: just run e-BH directly on the e-processes at each stopping time. The fix is a few lines, but the current proof needs replacing.\n\n- Corollary 4.2.1 repeats the same pattern. It claims FDR=1 at the first false detection time, which is only true if no true detections have occurred. The ARL bound E[tau*]≥1/alpha still holds, but from a direct e-detector threshold argument, not from FDR.\n\n- The abstract says \"under very general dependence structures on the data within and across the streams.\" That is over-claiming. The EOP guarantees need each e-detector to be valid w.r.t. the global filtration. Independent streams work, and some causal/past-dependence structures work, but Section 8.2's X_t^(2)=X_{t+1}^(1) example shows simple cross-stream dependence breaks it. The authors know this, but the abstract doesn't.\n\nNone of this touches the main EOP theorems or the impossibility results. For a multi-stream change detection audience this is a solid, useful paper. Worth a serious referee; I'd send it out after a revision that fixes Theorem 4.4's proof, corrects the corollary's argument, and softens the abstract.","headline":"Impossibility result and EOP metric are the real contributions; the e-d procedures prove the intended bounds, but one proof and one corollary need repair, and the abstract oversells the dependence generality.","tokens_in":33692,"tokens_out":10199,"would_cite":true,"duration_ms":94497,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62L10","60G40","62H15","62F03"],"pacs":[],"model":"deepseek-v4-flash","headline":"Any multi-stream change detector with finite average run length has trivial worst-case FDR, FWER, PFER, and GER, so the paper defines a new 'error over patience' metric and shows e-detector-based procedures can control it at any level.","keywords":["multi-stream change detection","multiple testing","error over patience","e-detectors","e-values","false discovery rate","average run length","anytime-valid inference"],"falsifier":"Run the paper's Section 8.2 counterexample with $K=2$ streams under the global null, setting $X^{(2)}_t = X^{(1)}_{t+1}$, and apply e-d-BH with any constant $\\alpha$; evaluating the FDR at the stopping time that waits until the first stream's e-detector crosses its threshold should violate the claimed EOP bound, because the global-filtration e-detector inequality fails in that configuration.","tokens_in":32570,"feed_emoji":"📉","tokens_out":5344,"duration_ms":43377,"temperature":0.7,"pith_summary":"This paper studies multi-stream sequential change detection, where several data streams are monitored for distributional changes and each declared change is a hypothesis test. It proves that any monitoring algorithm with finite average run length (the expected time to a false alarm under the global null) must have worst-case false discovery rate, family-wise error rate, per-family error rate, and global error rate equal to their trivial maximum values. Because a small detection delay forces a finite ARL, classical Type I error control is impossible in exactly the regime that matters in practice. The paper therefore defines a new family of metrics, error over patience (EOP), which divides the Type I error at a stopping time by the expected stopping time, and proposes e-d-BH, e-d-Bonferroni, e-d-Holm, and e-d-GNT, procedures that combine per-stream e-detectors with e-value multiple testing. These procedures control the EOP at any desired level, uniformly over all stopping times and change configurations, and control the classical metrics when the ARL is allowed to be infinite.","feed_headline":"Finite ARL makes worst-case FDR trivial in change detection","feed_subtitle":"A new error-over-patience metric is the only sensible target; e-value procedures control it at any level.","key_machinery":"The central objects are e-detectors, processes $(M_t^{(k)})_{t\\ge0}$ with $M_0=0$ that satisfy $\\mathbb{E}_P[M^{(k)}_\\tau]\\le\\mathbb{E}_P[\\tau]$ for every stopping time $\\tau$ under the null in stream $k$; they generalize CUSUM and Shiryaev-Roberts statistics and can be built from e-processes. The procedures feed the current e-detector values at each time $t$ into e-value multiple testing rules: e-BH (reject the largest $k^*$ streams with $M^{[k]}_t\\ge K/(k\\alpha_t)$), e-Bonferroni (reject streams with $M^{(k)}_t\\ge K/\\beta_t$), e-Holm, and a global-null test that rejects when $\\sum_k M^{(k)}_t\\ge K/\\alpha_t$. The proofs use the e-detector inequality at the stopped time $\\tau\\wedge(\\xi^{(k)}-1)$ to bound the expected number of false declarations by the expected patience $\\mathbb{E}[\\tau]$, and the e-BH/e-Bonferroni structure transfers the guarantee from a single time to all stopping times.","core_discovery":"The paper's central claim is that the Type I error metrics used in offline multiple testing cannot be nontrivially controlled in online multi-stream change detection whenever the algorithm has a finite ARL. For any such algorithm, under the global null, the first time to one detection is a stopping time at which every declaration is false, forcing worst-case FDR, FWER, PFER, and GER to their trivial values (1 for FDR, FWER, and GER, and at least the number of required detections for PFER). The correct target, the authors argue, is error over patience (EOP), defined as the supremum over stopping times of the ratio of a Type I error to the expected stopping time. The e-detector Benjamini-Hochberg (e-d-BH), e-detector Bonferroni (e-d-Bonferroni), and e-detector global-null (e-d-GNT) procedures control the EOP for FDR, PFER/FWER, and GER respectively at any level $\\alpha$, while the e-d-Holm procedure gives universal FWER control; all guarantees hold uniformly over stopping times and configurations when each stream's e-detector satisfies the e-detector inequality with respect to the global filtration.","pith_inferences":["The EOP ratio suggests a design principle for sequential monitoring that goes beyond this paper: whenever a finite expected stop time is essential, the natural Type I error target is the ratio of error to expected stop time, so the construction could transfer to online A/B testing or early stopping of clinical trials.","The paper's counterexample $X^{(2)}_t=X^{(1)}_{t+1}$ shows the global-filtration e-detector condition is not merely technical; practitioners should verify that no stream can 'see into the future' of another, since asynchronous sensors or look-ahead dependence can invalidate EOP control even when each stream's marginal detector is valid.","Because the impossibility result uses only Markov's inequality and the first-detection stopping time, it likely transfers to any error metric that equals 1 at the first false detection under the global null, implying that the EOP normalization is the natural companion to finite-ARL guarantees in a broad class of sequential multiple-testing problems."],"forward_implications":["Any multi-stream change detector that promises a finite ARL cannot simultaneously promise worst-case FDR, FWER, PFER, or GER below their trivial maximum values, regardless of how conservative its thresholds are.","The e-d-BH procedure with constant threshold $\\alpha$ bounds EOP$_{\\mathrm{FDR}}$ by $\\alpha$ and, under the global null, guarantees an ARL of at least $1/\\alpha$.","If finiteness of the ARL is forfeited, e-d-BH with $\\alpha_t=\\alpha/t$ controls FDR uniformly over all times and configurations, and e-d-Holm controls FWER uniformly under the same allowance.","The e-d-Bonferroni and e-d-GNT procedures provide the analogous EOP control for PFER/FWER and GER, again uniformly over all stopping times and configurations, under the global-filtration e-detector condition.","The EOP guarantees hold under arbitrary dependence within and across streams provided each stream's e-detector remains an e-detector for the global filtration; independent streams satisfy this condition automatically."],"supporting_citations":[{"why":"Supplies the e-detector framework and constructions (SR and CUSUM e-detectors) that the proposed procedures take as their input.","marker":"(Shin et al. (2023))"},{"why":"Provides the theory of e-processes and anytime-valid inference that underlies the definition and construction of e-detectors.","marker":"(Ramdas et al. (2023))"},{"why":"Gives the e-BH procedure and its FDR guarantee, which the e-d-BH algorithm extends to the sequential change detection setting.","marker":"(Wang and Ramdas (2022))"},{"why":"Gives the e-Bonferroni procedure and the averaging property of e-values used by the e-d-Bonferroni and e-d-GNT algorithms.","marker":"(Vovk and Wang (2021))"},{"why":"Defines the FDR metric that the paper shows is trivially maximal under a finite ARL and that the e-d-BH procedure controls in the EOP sense.","marker":"(Benjamini and Hochberg, 1995)"},{"why":"Provides the stepwise rejective procedure that the e-d-Holm algorithm adapts for universal FWER control in the sequential setting.","marker":"(Holm, 1979)"}],"fun_headline_variants":["Finite ARL makes worst-case FDR trivial; EOP is the fix","Classic error metrics fail in multi-stream detection; EOP works","E-detector procedures control error over patience at any level","Finite detection delay forces new metric: error over patience","Worst-case FDR impossible with finite ARL; e-values solve it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each stream's e-detector must satisfy the e-detector inequality for every stopping time that can use the global data filtration, so no stream may reveal information about another stream's future; this holds for independent streams but can fail under cross-stream dependence.","fun_headline_variants_meta":{"raw":{"variants":["Finite ARL makes worst-case FDR trivial; EOP is the fix","Classic error metrics fail in multi-stream detection; EOP works","E-detector procedures control error over patience at any level","Finite detection delay forces new metric: error over patience","Worst-case FDR impossible with finite ARL; e-values solve it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3123,"prompt_tokens":1046,"completion_tokens":2077,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1985}},"tokens_in":662,"tokens_out":2077,"duration_ms":15733,"temperature":1.0,"reasoning_tokens":1985,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:41:36.067402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's Section 8.2 counterexample with $K=2$ streams under the global null, setting $X^{(2)}_t = X^{(1)}_{t+1}$, and apply e-d-BH with any constant $\\alpha$; evaluating the FDR at the stopping time that waits until the first stream's e-detector crosses its threshold should violate the claimed EOP bound, because the global-filtration e-detector inequality fails in that configuration.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the FDR metric that the paper shows is trivially maximal under a finite ARL and that the e-d-BH procedure controls in the EOP sense."}],"review_version":1}