Pith. sign in

REVIEW 3 major objections 4 minor 4 references

Multiple testing in multi-stream sequential change detection

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Any multi-stream change detector with finite average run length has trivial worst-case FDR, FWER, PFER, and GER, so the paper defines a new 'error over patience' metric and shows e-detector-based procedures can control it at any level.

desk verdict Impossibility result and EOP metric are the real contributions; the e-d procedures prove the intended bounds, but one proof and one corollary need repair, and the abstract oversells the dependence generality. read the letter →

arxiv 2501.04130 v4 pith:7R3DTACB submitted 2025-01-07 math.ST eess.SPstat.MEstat.TH

Sanjit Dandapanthula , Aaditya Ramdas This is my paper · ORCID
classification math.STeess.SPstat.MEstat.TH MSC 62L1060G4062H1562F03
keywords multi-streamchangedetectionmultipletestingerroroverpatiencee-detectorse-valuesfalsediscoveryrateaveragerunlengthanytime-validinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies multi-stream sequential change detection, where several data streams are monitored for distributional changes and each declared change is a hypothesis test. It proves that any monitoring algorithm with finite average run length (the expected time to a false alarm under the global null) must have worst-case false discovery rate, family-wise error rate, per-family error rate, and global error rate equal to their trivial maximum values. Because a small detection delay forces a finite ARL, classical Type I error control is impossible in exactly the regime that matters in practice. The paper therefore defines a new family of metrics, error over patience (EOP), which divides the Type I error at a stopping time by the expected stopping time, and proposes e-d-BH, e-d-Bonferroni, e-d-Holm, and e-d-GNT, procedures that combine per-stream e-detectors with e-value multiple testing. These procedures control the EOP at any desired level, uniformly over all stopping times and change configurations, and control the classical metrics when the ARL is allowed to be infinite.

What carries the argument

The central objects are e-detectors, processes $(M_t^{(k)})_{t\ge0}$ with $M_0=0$ that satisfy $\mathbb{E}_P[M^{(k)}_\tau]\le\mathbb{E}_P[\tau]$ for every stopping time $\tau$ under the null in stream $k$; they generalize CUSUM and Shiryaev-Roberts statistics and can be built from e-processes. The procedures feed the current e-detector values at each time $t$ into e-value multiple testing rules: e-BH (reject the largest $k^*$ streams with $M^{[k]}_t\ge K/(k\alpha_t)$), e-Bonferroni (reject streams with $M^{(k)}_t\ge K/\beta_t$), e-Holm, and a global-null test that rejects when $\sum_k M^{(k)}_t\ge K/\alpha_t$. The proofs use the e-detector inequality at the stopped time $\tau\wedge(\xi^{(k)}-1)$ to bound the expected number of false declarations by the expected patience $\mathbb{E}[\tau]$, and the e-BH/e-Bonferroni structure transfers the guarantee from a single time to all stopping times.

What would settle it

Run the paper's Section 8.2 counterexample with $K=2$ streams under the global null, setting $X^{(2)}_t = X^{(1)}_{t+1}$, and apply e-d-BH with any constant $\alpha$; evaluating the FDR at the stopping time that waits until the first stream's e-detector crosses its threshold should violate the claimed EOP bound, because the global-filtration e-detector inequality fails in that configuration.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Type I error metrics used in offline multiple testing cannot be nontrivially controlled in online multi-stream change detection whenever the algorithm has a finite ARL. For any such algorithm, under the global null, the first time to one detection is a stopping time at which every declaration is false, forcing worst-case FDR, FWER, PFER, and GER to their trivial values (1 for FDR, FWER, and GER, and at least the number of required detections for PFER). The correct target, the authors argue, is error over patience (EOP), defined as the supremum over stopping times of the ratio of a Type I error to the expected stopping time. The e-detector Benjamini-Hochberg (e-d-BH), e-detector Bonferroni (e-d-Bonferroni), and e-detector global-null (e-d-GNT) procedures control the EOP for FDR, PFER/FWER, and GER respectively at any level $\alpha$, while the e-d-Holm procedure gives universal FWER control; all guarantees hold uniformly over stopping times and configurations when each stream's e-detector satisfies the e-detector inequality with respect to the global filtration.

Load-bearing premise

Each stream's e-detector must satisfy the e-detector inequality for every stopping time that can use the global data filtration, so no stream may reveal information about another stream's future; this holds for independent streams but can fail under cross-stream dependence.

Editorial extensions

If this is right

  • Any multi-stream change detector that promises a finite ARL cannot simultaneously promise worst-case FDR, FWER, PFER, or GER below their trivial maximum values, regardless of how conservative its thresholds are.
  • The e-d-BH procedure with constant threshold $\alpha$ bounds EOP$_{\mathrm{FDR}}$ by $\alpha$ and, under the global null, guarantees an ARL of at least $1/\alpha$.
  • If finiteness of the ARL is forfeited, e-d-BH with $\alpha_t=\alpha/t$ controls FDR uniformly over all times and configurations, and e-d-Holm controls FWER uniformly under the same allowance.
  • The e-d-Bonferroni and e-d-GNT procedures provide the analogous EOP control for PFER/FWER and GER, again uniformly over all stopping times and configurations, under the global-filtration e-detector condition.
  • The EOP guarantees hold under arbitrary dependence within and across streams provided each stream's e-detector remains an e-detector for the global filtration; independent streams satisfy this condition automatically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The EOP ratio suggests a design principle for sequential monitoring that goes beyond this paper: whenever a finite expected stop time is essential, the natural Type I error target is the ratio of error to expected stop time, so the construction could transfer to online A/B testing or early stopping of clinical trials.
  • The paper's counterexample $X^{(2)}_t=X^{(1)}_{t+1}$ shows the global-filtration e-detector condition is not merely technical; practitioners should verify that no stream can 'see into the future' of another, since asynchronous sensors or look-ahead dependence can invalidate EOP control even when each stream's marginal detector is valid.
  • Because the impossibility result uses only Markov's inequality and the first-detection stopping time, it likely transfers to any error metric that equals 1 at the first false detection under the global null, implying that the EOP normalization is the natural companion to finite-ARL guarantees in a broad class of sequential multiple-testing problems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies multiple-testing error control in multi-stream sequential change detection. The authors prove that any monitoring algorithm with finite average run length (ARL) has trivial worst-case FDR, FWER, PFER, and GER over all stopping times, motivating a new class of metrics called error-over-patience (EOP). They introduce four procedures—e-d-BH, e-d-Bonferroni, e-d-Holm, and e-d-GNT—that combine e-detectors with e-value multiple-testing methods, and prove EOP control as well as universal Type I error control when ARL finiteness is forfeited. The paper also discusses dependence assumptions, gives positive and negative examples, and reports simulations for parametric, nonparametric, and conformal change-detection settings.

Significance. The paper addresses a real gap: most multi-stream change-detection theory controls ARL or detection delay but not false-detection metrics across streams, while the multiple-testing literature does not treat sequential stopping. The impossibility results (Theorems 4.1, 5.1, Proposition 5.2, Proposition 7.1) are simple but important, and the EOP framework is a sensible response to the impossibility. The proposed algorithms are broadly applicable, come with explicit guarantees uniformly over stopping times, and are supported by reproducible simulations with provided code. The paper is transparent about the global-filtration assumption needed for EOP control and provides both positive and negative dependence examples. However, several proof details in the universal-control sections are incorrect as written, and the abstract overstates the dependence generality, so the paper needs revision before it can be accepted.

major comments (3)
  1. [Theorem 4.4] The proof of Theorem 4.4 is invalid in its current form. It argues that the event of any false detection is contained in the event that the BH procedure rejects a null when applied to p-values 1/(sup_t M_t). First, the standard BH procedure does not control FDR under arbitrary dependence among the p-values; it requires a condition such as PRDS, which is not established here. The correct tool is the e-BH theorem of Wang and Ramdas (2022), which controls FDR for arbitrarily dependent e-values. Second, replacing M_T by sup_t M_t does not preserve the e-variable property: an e-process need not have E[sup_t M_t] <= 1, so sup_t M_t is not guaranteed to be an e-variable. The theorem statement is likely true, but a correct proof should use the fact that the average (1/K) sum_k M_t is an e-process and apply Ville's inequality to bound the probability of any rejection across all t; the current proof needs to be rewritten.
  2. [Corollary 4.2.1 and Corollary 5.3.1] The proof of Corollary 4.2.1 asserts that FDR(phi, xi, tau*_1(phi, xi)) = 1 for arbitrary configurations xi. This is false when true and false detections occur at the same time: at the first false detection, the false discovery proportion can be strictly less than 1 if true detections are also present. A correct proof can be obtained from the inequality 1_{tau<infinity} <= sum_{k in H0} phi^{(k)}_tau and the e-detector property, which gives E[tau] >= 1/alpha; the current proof should be replaced. Similarly, the proof of Corollary 5.3.1 cites Theorem 4.2 instead of Theorem 5.3 and states PFER >= eta without conditioning; the statement itself is true when the ARL is finite, but the proof needs to be written correctly.
  3. [Abstract and Section 8] The abstract and introduction claim that the EOP guarantees hold 'under very general dependence structures' and 'under rather general settings and weak assumptions.' This overstates the actual condition: Theorems 4.2, 5.3, and 7.2 require each M^{(k)} to be an e-detector with respect to the global filtration F, not just the per-stream filtration. Section 8.2 gives a concrete example, X_t^{(2)} = X_{t+1}^{(1)}, where a per-stream e-detector is not a global-filtration e-detector, so the EOP theorems do not apply to that dependence structure. The paper is commendably transparent in Section 8, but the abstract and introduction should be qualified to say that EOP control holds under dependence structures that preserve the e-detector property under the global filtration (e.g., independent streams or the examples in Section 8.3), rather than under 'very general dependence.'
minor comments (4)
  1. [Algorithm 2, line 7] The condition in Algorithm 2 uses K/alpha_t, but the input parameter is (beta_t); it should be K/beta_t to match Theorem 5.3 and the proof.
  2. [Corollary 5.3.1 proof] The proof cites 'Theorem 4.2' when it should cite Theorem 5.3, and it writes ARL1(phi*, xi) with a star that is inconsistent with the notation elsewhere.
  3. [Theorem 4.4 proof, notation] The proof refers to 'step T' without defining T clearly; it appears to mean an arbitrary stopping time, but the notation should be made consistent with the theorem's tau.
  4. [Section 5.6 heading] The heading 'Controlling the worst-case PFER with a bounded PF A' should likely read 'PF_A' or 'probability of false alarm' to avoid confusion with PFER.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the EOP bounds are derived from the e-detector definition and explicitly stated algorithm thresholds, and the self-citations are background tools with independent proofs.

full rationale

The paper's central claims do not reduce to their inputs. In Theorem 4.2, the FDR bound is obtained by writing the e-d-BH rejection rule as an indicator threshold on M_t^(k) and applying the elementary inequality 1_{x>=1} <= x, then using the e-detector property E_P[M_tau] <= E_P[tau] at the stopping time tau ∧ (ξ(k)-1); the proof is shown in the paper and no fitted quantity enters. The same structure holds for Theorem 5.3 (PFER), Corollary 5.3.2 (FWER), and Theorem 7.2 (GER). The impossibility theorems (4.1, 5.1, 5.2, 7.1) are direct Markov-inequality arguments on the first detection time, not imported conclusions. The e-detector and e-process machinery is taken from Shin et al. (2023) and Ramdas et al. (2023), but those are prior results with stated assumptions and independent proofs; the paper's contribution is the composition with e-BH/e-Bonferroni, which is proven here. Section 8.2 explicitly shows a dependence structure (X_t^(2)=X_{t+1}^(1)) under which the global-filtration e-detector condition fails, so the scope of the EOP theorems is transparently stated rather than hidden. There is no fitted input relabeled as a prediction, and no equation is identical to its assumptions by construction. The only weakness is that the abstract's phrase 'very general dependence structures' is broader than the Section 8.1 condition, but that is a precision and risk issue, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The theoretical results have no fitted free parameters; thresholds are set analytically (alpha, beta), and simulation parameters (delta, signal strength) are illustrative inputs, not fitted to data. Appendix A describes an optional empirically chosen threshold c_alpha, but it is not needed for any theorem. No new physical entities are introduced; EOP is a metric, not an entity.

assumptions (4)
  • domain assumption For each stream k, the e-detector M^(k) is an e-detector with respect to the global filtration F, satisfying E_P[M^(k)_tau] <= E_P[tau] for all F-stopping times tau (Definition 2.12).
    This is the key assumption for Theorems 4.2, 5.3, and 7.2. Section 8 explains it holds for independent streams and for certain dependence structures, but fails when one stream sees another's future, as in the Section 8.2 counterexample.
  • domain assumption The e-BH and e-Bonferroni procedures control FDR and FWER/PFER respectively under arbitrary dependence among e-values, as established by Wang and Ramdas (2022) and Vovk and Wang (2021).
    The e-d-BH and e-d-Bonferroni algorithms inherit these guarantees at fixed times; this cited background is used implicitly in the proof of Theorem 4.4 and in the construction of the algorithms.
  • standard math Stopping times tau and tau ^ (xi(k) - 1) are F-stopping times, and xi(k) is a deterministic (unknown) configuration.
    Needed to apply the e-detector property at the stopped time tau ^ (xi(k) - 1) in Theorems 4.2, 5.3, and 7.2; true because xi is non-random and tau is an F-stopping time.
  • standard math Ville's inequality and Markov's inequality are valid for e-processes and nonnegative processes.
    Used in Theorems 4.1, 4.4, 5.5, and 7.4 to relate e-process thresholds to probabilities of false alarm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multiple testing in multi-stream sequential change detection." pith.science (2026). https://pith.science/paper/7R3DTACB

@misc{pith2026250104130,
  author       = {Pith},
  title        = {Pith review of: Multiple testing in multi-stream sequential change detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7R3DTACB}},
  note         = {Machine review of arXiv:2501.04130}
}
read the original abstract

Multi-stream sequential change detection involves simultaneously monitoring many streams of data and trying to detect when their distributions change, if at all. Here, we theoretically study multiple testing issues that arise from detecting changes in many streams. We point out that any algorithm with finite average run length (ARL) must have a trivial worst-case false detection rate (FDR), family-wise error rate (FWER), per-family error rate (PFER), and global error rate (GER); thus, any attempt to control these Type I error metrics is fundamentally in conflict with the desire for a finite ARL (which is typically necessary in order to have a small detection delay). One of our contributions is to define a new class of metrics which can be controlled, called error over patience (EOP). We propose algorithms that combine the recent e-detector framework (which generalizes the Shiryaev-Roberts and CUSUM methods) with the recent e-Benjamini-Hochberg procedure and e-Bonferroni procedures. We prove that these algorithms control the EOP at any desired level under very general dependence structures on the data within and across the streams. In fact, we prove a more general error control that holds uniformly over all stopping times and provides a smooth trade-off between the conflicting metrics. Additionally, if finiteness of the ARL is forfeited, we show that our algorithms control the worst-case Type I error.

Figures

Figures reproduced from arXiv: 2501.04130 by the authors.

Figure 1
Figure 1. SR e-detectors over time for detecting a Gaussian mean change at time 200 (y-axis is on a logarithmic scale). Now, suppose that ξ (k) = ∞ for k ∈ [K] = 1000, so that we are in the scenario of the global null. A naive change detection algorithm would declare a change in stream k when M (k) t ≥ 1/α. We find that e-d-BH (setting αt = α = 0.001) keeps the FDR at a much lower level than the naive algorithm, even in the l… view at source ↗
Figure 2
Figure 2. FDR of the naive algorithm and e-d-BH for Gaussian mean change. We know from Theorem 4.2 that e-d-BH theoretically keeps EOPFDR(φ) controlled at level α; in fact, we observe empirically that this translates to nontrivial control on the FDR. Similarly, although the naive 27 [PITH_FULL_IMAGE:figures/full_fig_p027_2.png] view at source ↗
Figure 3
Figure 3. FWER of the naive algorithm and e-d-Holm for Gaussian mean change (note the different y-axis scales). Finally, setting βt = β = 10 we see empirically that e-d-Bonferroni keeps the PFER controlled at a much lower level than the naive algorithm: 0 25 50 75 100 125 150 175 200 Time 0 100 200 300 400 500 600 700 P F E R(t) Naive PFER for Gaussian mean change (a) naive algorithm 0 25 50 75 100 125 150 175 200 Time 0 1 2 … view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: PFER of the naive algorithm and e-d-Bonferroni for Gaussian mean change (βt = β = 10). Next, suppose ξ (k) = 10 for k ∈ [K] = 1000 and we set αt = α = 0.001. We run the naive algorithm, e-d-BH, and e-d-Bonferroni 100 times, tracking the number of detections that they m…
Figure 5
Figure 5. Figure 5: Mean detections of e-d-BH, e-d-Bonferroni, and the naive algorithm over time for a Gaussian mean change (with varying signal strength). Hence, we see that our algorithms do not increase the detection delay too much, even compared to the naive algorithm. Despite the ver…
Figure 6
Figure 6. Figure 6: Mean detections of e-d-BH, e-d-Bonferroni, and the naive algorithm over time for a Gaussian mean change, with only 100 streams changing. Hence, in exchange for the conservatism of e-d-BH and e-d-Bonferroni, we make significantly fewer false detections. Note that there …
Figure 7
Figure 7. Figure 7: SR e-detectors over time for detecting a symmetry change at time 400. Next, we set αt = α = 0.001. We run the naive algorithm, e-d-BH, and e-d-Holm 100 times, tracking the number of detections that they make at each timestep. Here, we plot the mean number of detections…
Figure 8
Figure 8. Figure 8: Mean detections of e-d-BH, e-d-Holm, and the naive algorithm over time for a symmetry change. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Mean detections of e-d-BH, e-d-Holm, and the naive algorithm over time for a symmetry change (with only 10 streams changing). Again, we see that in exchange for their conservatism, e-d-BH and e-d-Holm make far fewer false detections than the naive algorithm. 9.3 Nonpar…
Figure 10
Figure 10. Figure 10: Conformal e-detectors over time for detecting a deviation from exchangeability at time 400. 3Vovk (2021) calls these betting functions instead of p-to-e calibrators. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 11
Figure 11. Figure 11: Mean detections of e-d-BH, e-d-Bonferroni, and the naive algorithm over time for non￾exchangeability. Recall that the number of detections increases, then decreases because as more data is collected after the change, it becomes less evident that the data seen thus far…
Figure 12
Figure 12. Figure 12: Mean detections of e-d-BH over time for symmetry changes at times t = 100 and t = 500. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_12.png]
Figure 13
Figure 13. Figure 13: e-detectors over time for detecting an independence change at time 400. Next, we set αt = α = 0.001. We run the naive algorithm, e-d-BH, and e-d-Holm 100 times, tracking the number of detections that they make at each timestep. Here, we plot the mean number of detecti…
Figure 14
Figure 14. Figure 14: Mean detections of e-d-BH, e-d-Bonferroni, and the naive algorithm over time for an independence change. As expected, the detection delay is larger for independence testing than for either Gaussian mean change detection or symmetry testing because independence testing…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

  1. [1]

    Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) , 57 (1), 289–

    work page 1995

  2. [4]

    Apply the dominated convergence theorem to the inequality in Proposition D.1 (since we know that CCD is uniformly bounded in [0 , 1]) as t → ∞

    37 Proof. Apply the dominated convergence theorem to the inequality in Proposition D.1 (since we know that CCD is uniformly bounded in [0 , 1]) as t → ∞. E Background on multiple testing with e-values Here, we discuss some of the procedures that inspired our algorithms. The e-BH (e-Benjamini Hochberg) procedure described in Wang and Ramdas (2022) provides...

    work page 2022

  3. [100]

    We call this piggybacking of evidence, since existing evidence for changes reduces the amount of evidence required to declare future changes

    However, it takes only 247 timesteps for e-d-BH to consistently detect all changes after the second subset of streams change at t = 500 (due to the decreased threshold for detection). We call this piggybacking of evidence, since existing evidence for changes reduces the amount of evidence required to declare future changes. C Simulation: nonparametric ind...

    work page 2023

  4. [300]

    Bodenham, D., & Adams, N. (2017). Continuous monitoring for changepoints in data streams using adaptive estimation. Statistics and Computing , 27, 1257–1270. Chan, H. P. (2017). Optimal sequential detection in multi-stream data. The Annals of Statistics . Chen, J., Zhang, W., & Poor, H. V. (2020). A false discovery rate oriented approach to parallel seque...

    arXiv 2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.