Online multi-layer FDR control

Ran Dai; Runqiu Wang

REVIEW 3 major objections 6 minor 17 references

Online multi-layer FDR control

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Three online testing procedures claim simultaneous false discovery rate control at every partition level of a streaming hypothesis list, under a per-layer conditional super-uniformity assumption.

desk verdict Genuinely new multi-layer extension of online FDR rules, but the central Theorem 2 proof is invalid as printed and Theorem 1's stated conditions don't match its own proof. read the letter →

arxiv 2506.03406 v1 pith:Z7X56ZXW submitted 2025-06-03 stat.ME

Runqiu Wang , Ran Dai This is my paper

classification stat.ME MSC 62L1062J1562F03

keywords onlinehypothesistestingfalsediscoveryratemodifiedFDRmulti-layerpartitionsalpha-investingLONDLORDsequential

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When hypotheses arrive one at a time and must be acted on immediately, online FDR rules such as LOND and LORD cap the fraction of false discoveries among rejected hypotheses. Real streams come with natural groupings—siRNA/nanocapsule combinations, user segments, gene sets—and decisions are wanted at several grouping levels at once. This paper claims the online machinery works layer by layer: multi-layer α-investing controls the modified FDR (expected false discoveries divided by expected discoveries plus a constant) at every layer simultaneously, and multi-layer LOND and LORD control both FDR and modified FDR at every layer, with a hypothesis rejected only when every still-pending layer's threshold is passed. If correct, a screening pipeline can report simultaneous false-discovery guarantees for every fixed partition of the stream without collecting batches or waiting for it to end.

What carries the argument

The carrying object is the per-layer decision-and-counting process. Each layer m keeps its own threshold rule: LOND sets $\alpha^m_i = \beta^m_i(R^m(i-1)+1)$ with $\sum_i \beta^m_i = \alpha$; LORD sets $\alpha^m_i = \beta^m_{\zeta^m_i}$, where $\zeta^m_i$ counts steps since the layer's last discovery and resets to 1 at every discovery; α-investing updates a wealth $W^m(i)$ by spending $\phi^m_i$ and rewarding $\psi^m_i$ on rejection. The conjunction rule makes the individual decision $\delta_i = 1$ only if, for every layer m whose group $g_i^m$ is still pending, the layer-specific p-value $p_i^m$ is below $\alpha^m_i$. For α-investing the proof is carried by the per-layer submartingale $A^m(j) = \alpha R^m(j) - V^m(j) + \alpha\eta - W^m(j)$; for LOND and LORD the proofs telescope the per-layer thresholds so that the total level spent between discoveries is at most $\alpha$.

What would settle it

Construct a two-layer stream in which layer-2 null p-values are generated conditional on layer-1 decisions—for example, drawn small only after layer 1 rejects—so the conjunction rule can inflate the layer-2 rejection rate; compute $P(\delta^m_{g^m_j}(j)=1 \mid F_{j-1})$ for pending groups and run repeated trials. If the conditional probability exceeds the assigned threshold $\alpha^m_j$, or if the empirical layer-2 FDR exceeds $\alpha$, the key bound behind Theorems 2 and 3 is violated.

Watch

Extended reading notes

Core claim

The paper's central claim is that simultaneous multi-layer FDR control is achievable in the fully online setting, with group memberships revealed one hypothesis at a time and not fixed in advance. For each of M fixed partitions (layers) of the hypothesis indices, the procedure maintains its own sequential testing rule—a wealth process in α-investing, a discovery-count sequence in LOND, a discovery-renewed level in LORD—and rejects an individual hypothesis H_i only if, for every layer m whose group g_i^m has not already been rejected, its layer-specific p-value p_i^m falls below that layer's current threshold. A layer-m group counts as selected once any member hypothesis is selected, and a true null group in layer m is one with no true alternative among its members; layer-m FDR and mFDR are computed on these group-level sets. Theorems 1–3 state that under Assumption 1—conditional null rejection probability for a pending group bounded by the assigned threshold—multi-layer α-investing controls mFDR at all layers, while multi-layer LOND and LORD control both FDR and mFDR at all layers for every finite horizon n. The proofs are built from per-layer submartingales for α-investing and from threshold-summation arguments for LOND and LORD.

Load-bearing premise

For every layer and every true null hypothesis, the chance that the layer's group is rejected given everything that happened before must be no larger than the significance level that layer's rule assigns, and this must remain true even when a hypothesis is rejected only after every still-pending layer passes its test.

Editorial extensions

If this is right

A screening experiment can split its stream by any number of fixed partitions and quote one overall $\alpha$ for every layer at once, including the individual-hypothesis layer as a special case.
Because the rules are online, decisions remain valid if the stream is unbounded; no prespecified total number of hypotheses or group sizes is needed.
The multi-layer rules preserve the individual-level FDR guarantee of their single-layer ancestors, so moving to multi-layer screening does not sacrifice the usual per-comparison guarantee.
The paper's simulations indicate the multi-layer versions restore group-level FDR control where the unmodified rules fail, for example α-investing under block-structured streams and LORD under random signal patterns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same per-layer conjunction architecture should extend to other online error metrics, such as FWER or FDR under arbitrary dependence, provided a layer-wise rule with an analogous conditional super-uniformity exists; the paper proves only the three named rules.
Editorial inference: the modified multi-layer LOND, which re-indexes by the number of tests a layer actually faces, improves power in the paper's simulations but has no theorem; the paper itself flags its theoretical guarantee as future work.
Editorial inference: because the guarantees require fixed partitions and fixed group memberships, an operator who adaptively redefines layers after seeing the stream steps outside the theorem's scope.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real, useful extension—simultaneous FDR control across several partitions of an online hypothesis stream—and the algorithms are natural. But the central proof has a load-bearing error, and the paper is not ready as is.

What's new: the multi-layer α-investing, LOND, LORD procedures, with group-level decisions that require all pending layer tests to pass. That is a sensible way to get simultaneous guarantees. The paper is right that the setting (online, group membership not predefined) is not obviously covered by p-filter or DAGGER, though a sharper comparison would help. Simulations cover block, interleaved, and unbalanced group arrivals, plus several signal patterns and signal-strength settings; the multi-layer methods behave as advertised.

The soft spots are in the theory. The proof of Theorem 2 replaces V^m(n) with the sum over j of δ^m_{g^m_j}(j), then invokes the null-only conditional bound E[δ/α | ...] ≤ 1 for every j. That step is not valid: for alternatives the bound is unavailable, and once a group is rejected, δ stays 1 for every later member, so the same discovery is counted multiple times with conditional probability 1. The displayed inequality does not follow from the theorem's condition. The same issue appears in the mFDR part, where E(δ−α) ≤ 0 is applied to all hypotheses. The simultaneous FDR guarantee is therefore unproven as printed, though the claim may still be true and likely repairable by a first-rejection-time argument.

Theorem 1 has a separate inconsistency. The proof of Lemma 1 needs ψ^m_j ≤ φ^m_j/α^m_j + α − 1 in the first null case, but the theorem states ψ^m_j ≤ φ^m_j/α^m_j + α + 1. The condition in the theorem is looser than the algebra supports.

No code is shipped, so the simulations are not independently reproducible; minor. The paper also explicitly omits the proof of the improved LOND variant, which is acceptable if flagged.

Conclusion: the idea is good, the writing is clear, and the simulations are consistent. If the proofs are fixed, this is a publishable subfield contribution. As is, it should not be accepted without revision.

Recommendation: send to peer review, but warn the referee that the Theorem 2 proof needs to be rewritten from first principles.

Referee Report

3 major / 6 minor

Summary. The paper proposes three online hypothesis-testing procedures for settings where hypotheses arrive sequentially and are partitioned into groups according to several alternative structures ('layers'). The methods are multi-layer versions of α-investing, LOND, and LORD. The main theoretical claims are that multi-layer α-investing simultaneously controls mFDR at every layer, and multi-layer LOND and LORD simultaneously control both FDR and mFDR at every layer, under conditional super-uniformity assumptions on group-level decisions. Numerical simulations illustrate FDR, mFDR, and power across several group structures, signal patterns, and signal strengths.

Significance. If the theoretical claims are correct, the paper addresses a genuine gap: simultaneous false-discovery guarantees for multiple online partitions of a hypothesis stream, which is relevant to drug screening, genomics, and A/B testing. The problem is well motivated and the proposed algorithms are natural extensions of established online FDR methods. However, the central proof for multi-layer LOND contains a load-bearing invalid step, and Theorem 1's stated reward condition is inconsistent with its proof. As a result, the principal theoretical contributions are not established as printed. The paper has useful simulation evidence but requires substantial revision to justify its claims.

major comments (3)

[Section 4, proof of Theorem 2 (multi-layer LOND)] The proof of Theorem 2 is invalid as printed. The first displayed inequality bounds V^m(n) by Σ_{j=1}^n δ^m_{g^m_j}(j); the subsequent step asserts E[δ^m_{g^m_j}(j)/α^m_j | ...] ≤ 1 for every j. However, Theorem 2's condition applies only when θ_j=0 and δ^m_{g^m_j}(j−1)=0. For alternative hypotheses and for all members of a group after its first rejection, δ^m_{g^m_j}(j) is either not covered by the assumption or is deterministically 1, so the claimed bound does not follow. The same error vitiates the mFDR part, where E(δ^m−α^m)≤0 is applied to all hypotheses. Consequently the simultaneous FDR and mFDR guarantees for multi-layer LOND are not established. A correct argument would need to work with the first rejection times of null groups and use the pending-group condition only at those times.
[Section 3.1 / Theorem 1 and Web Appendix A] Theorem 1 states the reward condition 0≤ψ^m_j≤min(φ^m_j/ρ^m_j+α, φ^m_j/α^m_j+α+1), but the proof of Lemma 1 uses the stricter bound ψ^m_j≤φ^m_j/α^m_j+α−1 in both the first and second cases. With the printed +1, the displayed lower bounds in the proof can be negative, so the theorem's sufficient condition is inconsistent with its proof. The correct condition (or a corrected proof) must be supplied before the mFDR control claim for multi-layer α-investing can be accepted.
[Section 4, Theorem 3 and its proof] The mFDR part of Theorem 3 begins with 'Eθ(V^m(n)) ≤ Σ_{i=1}^n Eθα^m_i' without justification; V^m(n) counts rejected false groups, not individual rejections, and bounding it requires the same first-rejection argument that is missing in Theorem 2. In the FDR part, the proof's notation is not self-consistent: the index i is used both for the discovery number and for the time l, and the indicator I(ζ^m_i ≠ ζ^m_{i−1}) appears where I(ζ^m_l ≠ ζ^m_{l−1}) is needed. As printed, the proof of Theorem 3 is incomplete.

minor comments (6)

[Throughout] The manuscript contains many typographical and formatting errors, such as duplicated citations ('Foster & Stine (2008)Foster & Stine (2008)'), misspellings ('mili-layer', 'mulit-layer'), and inconsistent method names in Section 5.4 ('modified multi-layer LORD' versus ml-LORD / ml-LOND_m). A thorough proofreading is needed.
[Section 4, Theorem 2 statement] The assumption in Theorem 2 begins with '∀θ_i∈Θ,∀m=1,...,M,' followed by a probability subscripted θ_i=0; this is confusing and should be rewritten to state explicitly that the bound is required only for null individual hypotheses (θ_i=0) while the group is pending.
[Abstract and Section 4] The abstract and introduction claim simultaneous FDR control, but Theorem 3 only establishes control of FDR at discovery times (and mFDR_1), not FDR at arbitrary times n. The authors should state precisely in the summary which quantity is controlled at which time horizon.
[Section 4, Assumption 1 and Theorem 2 condition] The paper does not explain how the conditional super-uniformity of group decisions follows from standard super-uniformity of the underlying p-values under the multi-layer conjunction rule. A short remark connecting the two would help readers verify the assumption in applications.
[Section 3.2, Remark after Algorithm 2] The remark states that the improved LOND method's theory is omitted but its power is shown by simulation; the simulations indeed use ml-LOND_m. This is acceptable, but the text should not imply that Theorem 2 covers the modified method, since it does not.
[Figures and captions] The figure captions are incomplete: for example, Figure 1 says '(c) Unbalanced Structure' but the panel labels in the caption and text are not consistently provided. Please ensure all panels are clearly labeled and referenced.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the FDR/mFDR guarantees are derived from explicit conditional p-value bounds, not built into the definitions or fitted from the data.

full rationale

The paper's central results (Theorems 1-3) are not circular. Each theorem starts from an explicit probabilistic assumption on null rejection probabilities given the available history: Assumption 1 for alpha-investing, the conditional superuniformity condition in Theorem 2 for LOND, and the analogous condition in Theorem 3 for LORD. These assumptions are about single-step conditional probabilities of rejecting a null group, not about the target FDR or mFDR quantities. The conclusions are then obtained by summation, optional stopping, and submartingale arguments that use the predefined beta sequences and the assumed bounds. The beta sequences and the wealth parameter eta are user-chosen tuning inputs, not fitted constants, and no 'prediction' is formed by renaming a fitted quantity. The paper also does not rely on load-bearing self-citations: its references are to prior external works (Foster and Stine, Javanmard and Montanari, Ramdas et al.), and the actual proofs are provided in the appendix rather than imported from an unpublished source. The acknowledged omission of a theoretical guarantee for the modified multi-layer LOND variant is a limitation, not circular reasoning. Overall, no step in the claimed derivation reduces by construction to its own inputs, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No parameters are fitted to data in the theoretical claims. The per-layer beta sequences and the eta offset are user-selected tuning constants; the theorems hold for any valid choice. The main assumptions are the per-layer conditional super-uniformity condition (Assumption 1 / Theorem 2 condition), the group null and rejection definitions, and standard martingale arguments. There are no invented entities.

free parameters (2)

per-layer significance sequences beta^m_j = user-specified, sum over j equals alpha
LOND, LORD and their multi-layer versions require one nonnegative sequence per layer summing to alpha; the theorems hold for any such sequence, so this is a tuning choice, not a fitted parameter.
eta (mFDR offset and initial wealth scale) = user-specified, positive
Appears in the definition mFDR^m_eta = E[V] / (E[R] + eta) and in the initial wealth W^m(0) = alpha eta; not fitted to data.

assumptions (3)

domain assumption Assumption 1: for null hypotheses, P(delta^m_{g_j}(j)=1 | F_{j-1}) <= alpha^m_j for every layer m, and for alternatives P <= rho^m_j.
This conditional super-uniformity is the engine of all three theorems; it is assumed, not derived from primitive p-value distributions.
domain assumption Group-level null is defined as all members null (intersection) and group-level rejection as at least one member rejected (union).
This is the paper's definition of the target error metric; if the application requires a different group-level truth rule, the guarantees do not apply.
standard math Standard probability and martingale results, including optional stopping for nonnegative supermartingales and submartingales.
Used in the proofs of Lemma 1 and Theorems 1 through 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Online multi-layer FDR control." pith.science (2026). https://pith.science/paper/Z7X56ZXW

@misc{pith2026250603406,
  author       = {Pith},
  title        = {Pith review of: Online multi-layer FDR control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7X56ZXW}},
  note         = {Machine review of arXiv:2506.03406}
}

read the original abstract

When hypotheses are tested in a stream and real-time decision-making is needed, online sequential hypothesis testing procedures are needed. Furthermore, these hypotheses are commonly partitioned into groups by their nature. For example, the RNA nanocapsules can be partitioned based on therapeutic nucleic acids (siRNAs) being used, as well as the delivery nanocapsules. When selecting effective RNA nanocapsules, simultaneous false discovery rate control at multiple partition levels is needed. In this paper, we develop hypothesis testing procedures which controls false discovery rate (FDR) simultaneously for multiple partitions of hypotheses in an online fashion. We provide rigorous proofs on their FDR or modified FDR (mFDR) control properties and use extensive simulations to demonstrate their performance.

Figures

Figures reproduced from arXiv: 2506.03406 by the authors.

**Figure 2.** Power, FDR and mFDR for individual and group layers for different [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

**Figure 3.** Power, FDR and mFDR for individual and group layers for different [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Power, FDR and mFDR for individual and group layers for different [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

B., Wang, Y., Barber, R

Berrett, T. B., Wang, Y., Barber, R. F., and Samworth, R. J. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 0 (1): 0 175--197, 2020

work page 2020
[3]

B., Dhumane, J

Deore, A. B., Dhumane, J. R., Wagh, R., and Sonawane, R. The stages of drug discovery and development process. Asian Journal of Pharmaceutical Research and Development, 7 0 (6): 0 62--67, 2019

work page 2019
[4]

S., Nazari, A., Momayezi, A., Faghihkhorasani, F., Abdulwahid, A.-H

Ebrahimi, N., Manavi, M. S., Nazari, A., Momayezi, A., Faghihkhorasani, F., Abdulwahid, A.-H. R. R., Rezaei-Tazangi, F., Kavei, M., Rezaei, R., Mobarak, H., et al. Nano-scale delivery systems for sirna delivery in cancer therapy: New era of gene therapy empowered by nanotechnology. Environmental research, 239: 0 117263, 2023

work page 2023
[5]

Foster, D. P. and Stine, R. A. -investing: a procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70 0 (2): 0 429--444, 2008

work page 2008
[6]

Goeman, J. J. and B \"u hlmann, P. Analyzing gene expression data in terms of gene sets: methodological issues. Bioinformatics, 23 0 (8): 0 980--987, 2007

work page 2007
[7]

and Montanari, A

Javanmard, A. and Montanari, A. On online control of false discovery rate. arXiv preprint arXiv:1502.06197, 2015

arXiv 2015
[8]

and Montanari, A

Javanmard, A. and Montanari, A. Online rules for control of false discovery rate and false discovery exceedance. The Annals of statistics, 46 0 (2): 0 526--554, 2018

work page 2018

Show all 17 references

[9]

Recent developments in nanoparticle-based sirna delivery for cancer therapy

Lee, J.-M., Yoon, T.-J., and Cho, Y.-S. Recent developments in nanoparticle-based sirna delivery for cancer therapy. BioMed research international, 2013 0 (1): 0 782041, 2013

2013
[10]

Saffron: an adaptive algorithm for online control of the false discovery rate

Ramdas, A., Zrnic, T., Wainwright, M., and Jordan, M. Saffron: an adaptive algorithm for online control of the false discovery rate. In International conference on machine learning, pp.\ 4286--4294. PMLR, 2018

2018
[11]

J., and Jordan, M

Ramdas, A., Chen, J., Wainwright, M. J., and Jordan, M. I. A sequential algorithm for false discovery rate control on directed acyclic graphs. Biometrika, 106 0 (1): 0 69--86, 2019 a

2019
[12]

K., Barber, R

Ramdas, A. K., Barber, R. F., Wainwright, M. J., and Jordan, M. I. A unified treatment of multiple testing with prior knowledge using the p-filter. The Annals of Statistics, 47 0 (5): 0 2790--2821, 2019 b

2019
[13]

Heterogeneous treatment effects in digital experimentation

Taddy, M., Gardner, M., Chen, L., and Draper, D. Heterogeneous treatment effects in digital experimentation. arXiv preprint arXiv:1412.8563, 2015

2015 arXiv
[14]

and Ramdas, A

Tian, J. and Ramdas, A. Online control of the familywise error rate. Statistical methods in medical research, 30 0 (4): 0 976--993, 2021

2021
[15]

False discovery rate controlled heterogeneous treatment effect detection for online controlled experiments

Xie, Y., Chen, N., and Shi, X. False discovery rate controlled heterogeneous treatment effect detection for online controlled experiments. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp.\ 876--885, 2018

2018
[16]

Online control of the false discovery rate in group-sequential platform trials

Zehetmayer, S., Posch, M., and Koenig, F. Online control of the false discovery rate in group-sequential platform trials. Statistical Methods in Medical Research, 31 0 (12): 0 2470--2485, 2022

2022
[17]

D., Kuan, P

Zhang, X. D., Kuan, P. F., Ferrer, M., Shu, X., Liu, Y. C., Gates, A. T., Kunapuli, P., Stec, E. M., Xu, M., Marine, S. D., et al. Hit selection with false discovery rate control in genome-scale rnai screens. Nucleic acids research, 36 0 (14): 0 4667--4679, 2008

2008

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

B., Wang, Y., Barber, R

Berrett, T. B., Wang, Y., Barber, R. F., and Samworth, R. J. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 0 (1): 0 175--197, 2020

work page 2020

[3] [3]

B., Dhumane, J

Deore, A. B., Dhumane, J. R., Wagh, R., and Sonawane, R. The stages of drug discovery and development process. Asian Journal of Pharmaceutical Research and Development, 7 0 (6): 0 62--67, 2019

work page 2019

[4] [4]

S., Nazari, A., Momayezi, A., Faghihkhorasani, F., Abdulwahid, A.-H

Ebrahimi, N., Manavi, M. S., Nazari, A., Momayezi, A., Faghihkhorasani, F., Abdulwahid, A.-H. R. R., Rezaei-Tazangi, F., Kavei, M., Rezaei, R., Mobarak, H., et al. Nano-scale delivery systems for sirna delivery in cancer therapy: New era of gene therapy empowered by nanotechnology. Environmental research, 239: 0 117263, 2023

work page 2023

[5] [5]

Foster, D. P. and Stine, R. A. -investing: a procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70 0 (2): 0 429--444, 2008

work page 2008

[6] [6]

Goeman, J. J. and B \"u hlmann, P. Analyzing gene expression data in terms of gene sets: methodological issues. Bioinformatics, 23 0 (8): 0 980--987, 2007

work page 2007

[7] [7]

and Montanari, A

Javanmard, A. and Montanari, A. On online control of false discovery rate. arXiv preprint arXiv:1502.06197, 2015

arXiv 2015

[8] [8]

and Montanari, A

Javanmard, A. and Montanari, A. Online rules for control of false discovery rate and false discovery exceedance. The Annals of statistics, 46 0 (2): 0 526--554, 2018

work page 2018