{"id":"25d98a64-0f41-4e98-a995-c0df1e3eff27","arxiv_id":"1908.05319","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"sGBH selects groups of hypotheses of interest and applies weighted FDR control only to those groups, with conservativeness conditions and improved empirical power over GBH in sparse configurations.","lead":"This statistics paper proposes sGBH, a two-stage procedure for false discovery rate control that first picks groups of hypotheses likely to contain real effects, then applies weighted Benjamini-Hochberg testing only to those groups. It reports FDR control conditions and simulations showing more power than the standard grouped GBH procedure when most groups are pure noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-dependent grouping in Section 5 invalidates the FDR guarantee: threshold-based p-value bins make nulls sub-uniform, so Theorem 2 does not cover the reported application.","rationale":"The reader's weakest assumption correctly identifies the central gap. The paper's theoretical engine, Theorem 2 plus Lemma 1, establishes FDR control for a procedure whose group partition is fixed before testing and whose null p-values are super-uniform. In the Section 5 application, however, the groups are defined by thresholds on the very p-values being tested, so the null p-values within the tested group H2 are sub-uniform (truncated uniforms on [0.15, 0.7]). This directly breaks the super-uniformity inequality used in the proof of Theorem 2. The data application is not a minor detail: it is the headline empirical demonstration and the source of the reported improvement (759 vs 487 rejections). Because the authors caution that assumptions may be violated but still present the improvement without a corrected analysis, the manuscript needs a revision: either re-run the application with a pre-specified partition (e.g., based on external gene annotations or expression variability) or extend the theory to cover data-dependent grouping. Neither requires rejecting the paper's theoretical contribution, so the CONDITIONAL verdict stands unchanged. The proposed permutation test would settle whether the data-dependent grouping actually inflates FDR, and it is straightforward to implement with the existing code and data.","tokens_in":37407,"tokens_out":5408,"duration_ms":57838,"concrete_test":"Generate null data calibrated to the prostate study by permuting case/control labels B=1000 times (preserving gene-gene dependence). For each permutation, apply the exact Section 5 pipeline: compute two-sided t-test p-values for all genes; partition the 4374 genes into H1 (p>0.7), H2 (0.15<p<0.7), H3 (p<0.15); select S by Simes at level 0.1; estimate groupwise proportions with Jin's estimator using the Section 5 gamma; run plug-in adaptive sGBH at alpha=0.05; record the false discovery proportion. If the average FDP exceeds 0.05 by more than Monte Carlo error, the data-dependent partition breaks FDR control. As a control, repeat with a partition fixed before seeing p-values, e.g., randomly assigning genes to three groups of the same sizes; the same pipeline should control FDR if the violation is specifically the threshold-based grouping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The procedure's FDR control rests on the fixed-partition and super-uniformity assumptions. Section 3 states the partition {G_j} is known before seeing p-values, and Section 2 assumes every null p-value is super-uniform, i.e., F_i(t) <= t. Theorem 2's proof (Appendix A.3, around Eq. (26)) uses exactly this inequality to bound E[1{p_jk <= rα/(c_jk m_S)} | p_-jk] by rα/(c_jk m_S). Section 5 constructs the partition from the same p-values to be tested: H1 has p>0.7, H2 has 0.15<p<0.7, H3 has p<0.15. For a true null in H2, the conditional distribution is not super-uniform: F(t) = (t-0.15)/0.55 on (0.15,0.7), so F(0.6)=0.818>0.6. Consequently the key inequality in the proof of Theorem 2 fails, and the plug-in adaptive sGBH as actually run on the prostate data is not covered by the paper's theory. The authors themselves caution at the end of Section 5 that FDRs may exceed the nominal level, but the reported 759 vs 487 improvement is still presented without a valid FDR guarantee. This is the load-bearing gap: the central asymptotic/non-asymptotic FDR claim applies to a different procedure than the one demonstrated on the data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a two-stage grouped FDR procedure (sGBH) for testing structured hypotheses. In the first stage, groups of hypotheses judged to contain signals (the \"interesting\" groups) are selected; in the second stage, only p-values in those groups are weighted by estimated groupwise null proportions and subjected to BH. The authors prove that the oracle sGBH coincides with the oracle GBH under sparse configurations (Theorem 1), give an FDR upper bound for the plug-in adaptive sGBH under conditions of reciprocal conservativeness and consistency (Theorem 2), provide a KS-based consistency lemma for selecting interesting groups (Lemma 1), and report simulations and a prostate cancer data application suggesting power gains over adaptive GBH.","tokens_in":37694,"tokens_out":9998,"duration_ms":93656,"significance":"The theoretical results are presented with explicit conditions rather than assuming the conclusion; Theorem 2's non-asymptotic bound honestly includes the costs of estimating S and the null proportions, and the proof strategy via reciprocal conservativeness is sound and generalizes earlier work. If the FDR guarantee held for the demonstrated application, the method would give practitioners a useful alternative to GBH when most groups are null. The potential power improvement is supported by simulations. However, the practical demonstration in Section 5 is not covered by the stated theory, and the simulation setting for the plug-in version with Jin's estimator is not fully covered either; these gaps substantially reduce the strength of the empirical claims.","major_comments":[{"comment":"The application constructs the groups H1, H2, H3 by thresholding the same p-values that are later tested (p>0.7, 0.15<p<0.7, p<0.15), with H2 and H3 randomly subsampled. Section 3 explicitly assumes the partition {G_j} is known before seeing the p-values, and Section 2 assumes every null p-value is super-uniform. For a true null p-value in H2, the conditional distribution given 0.15<p<0.7 is F(t)=(t-0.15)/0.55, so F(0.6)=0.818>0.6; it is not super-uniform. Therefore the key inequality used in the proof of Theorem 2 (Appendix A.3, around Eq. (26)) does not hold for the procedure as run, and Proposition 1 also does not apply. The reported comparison of 759 vs 487 rejections is thus not backed by an FDR guarantee. The caution at the end of Section 5 acknowledges this, but the section still presents the improvement without a valid theoretical basis. Please either use a pre-specified partition based on external information, extend the theory to threshold-based data-dependent partitions, or reclassify the application as exploratory without FDR claims.","section":"Section 5 (Application)"},{"comment":"The plug-in adaptive sGBH in the main simulations uses Jin's estimator to estimate groupwise null proportions, but Theorem 2's non-asymptotic bound requires each \\hat{\\pi}_{j0} to be non-increasing and reciprocally conservative; no such property is proved for Jin's estimator. The asymptotic part of Theorem 2 requires PRDS, and Section 4.1 notes that two-sided p-values \"may not\" satisfy PRDS. Consequently the simulation results with Jin's estimator and two-sided p-values are not covered by either arm of Theorem 2. The text in Section 4.2 stating that the adaptive sGBH \"is conservative\" is therefore an empirical observation, not a consequence of the proved theory; please state this distinction explicitly, or verify the required conditions in the simulations.","section":"Section 4 (Simulation study)"},{"comment":"The application uses Simes test to select \\hat{S}, and the generic adaptive sGBH simulations of Section 4 use Simes at various \\xi. Theorem 2's consistency condition for \\hat{S} is supplied by Lemma 1 for the KS test only; the paper does not prove that Simes-based selection is consistent under the sparse configurations considered. If Simes is to be used in the main procedure, a consistency result (or a simulation-based demonstration that selection errors vanish) is needed to bring the application under Theorem 2.","section":"Section 3.1 and Section 5"}],"minor_comments":[{"comment":"The inequality \\hat{\\pi}^{\\dagger}_{0,k} \\le \\hat{\\pi}^{\\dagger}_0 stated immediately after (5) is reversed for a non-increasing estimator: setting a p-value to 0 should not decrease the estimate. The correct inequality is \\ge, and the proof of Theorem 2 actually relies on this corrected direction.","section":"Definition 2"},{"comment":"Condition (8) is a limit of random quantities; please state the mode of convergence (in probability, almost surely) or replace it by a high-level assumption that the KS test is consistent, as the proof uses it to conclude Pr(\\hat{S}=S)\\to 1.","section":"Lemma 1"},{"comment":"The random sampling of 1500 genes from p-value intervals makes the partition random as well as threshold-based; even if the threshold-based grouping were pre-specified, the random subsampling adds another layer of data dependence that is not present in the theoretical setup.","section":"Section 5"},{"comment":"There are small typographical errors: \"mutliple\" in the Benjamini and Yekutieli reference, and \"there exits a constant\" in Theorem 4 of the supplementary material.","section":"References and supplementary material"}],"recommendation":"major_revision","confidential_remarks":"The data-dependent grouping issue in Section 5 is the most serious problem; if the authors cannot provide a valid FDR guarantee for this particular analysis, they should present it as a case study of the procedure's operating characteristics rather than as an FDR-controlled discovery list. The core proofs appear sound, so the path to revision is clear: either bring the application under the theory or soften the claims. No novelty disclosure concerns; this is a methodological extension with useful potential."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine, useful refinement of GBH for sparse group structures, and the main theorems are technically sound. But the data application builds groups by thresholding the very p-values being tested, which violates the fixed-partition and super-uniformity assumptions the theory relies on. The reported 759 vs 487 gain is therefore not backed by the paper's FDR guarantee. That gap is fixable, and the rest of the paper deserves a serious referee.\n\nWhat's new: the two-stage idea—select groups of interest, then weight only those groups—is a sensible extension of GBH and TLTA. The conditions for conservativeness (reciprocally conservative estimators, consistency of S) and the FDR bound that explicitly accounts for group-selection error are original and useful. Theorem 1's equivalence between oracle sGBH and oracle GBH is cleanly proved, and Theorem 2's bound looks internally consistent. The simulations are extensive for both independence and AR dependence, and the paper is honest about the empiric nature of the power advantage.\n\nSoft spots: the load-bearing one is Section 5. The paper states at the start of Section 3 that the partition is known and fixed before seeing p-values. The application partitions hypotheses by thresholds on the same p-values (p>0.7, 0.15<p<0.7, p<0.15). Under the null, a p-value in (0.15,0.7) is not super-uniform—its conditional distribution has F(0.6)=0.818>0.6—so the key inequality in the proof of Theorem 2 fails. The authors do caution that FDRs may exceed the nominal level, but then they still present the 759 vs 487 comparison as if the procedure controls FDR. That's a mismatch between theory and practice. It doesn't invalidate the theoretical contribution, but it means the application should be recast as exploratory, or the groups should be pre-specified from external information.\n\nA secondary, minor issue: the claimed power improvement over adaptive GBH is demonstrated empirically, not theoretically. The paper says this clearly, so it's not a hidden flaw.\n\nBottom line: this is a worthwhile paper for anyone working on structured FDR control. The theory for fixed partitions is solid; the application overreaches. I'd send it to peer review with a requested major revision to either fix the application's assumptions or scale back its claims.","headline":"Useful refinement of grouped FDR control with solid theory for fixed partitions, but the real-data application breaks the paper's own assumptions; the gap is fixable.","tokens_in":38261,"tokens_out":2759,"would_cite":true,"duration_ms":25292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J15","62F03","62F05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that a two-stage, selectively weighted FDR procedure controls the false discovery rate when only some groups contain real effects, provided groups are fixed beforehand and null proportions are conservatively estimated.","keywords":["false discovery rate","grouped hypotheses","weighted p-values","adaptive procedure","sparse configuration","proportion of true nulls","multiple testing","reciprocal conservativeness"],"falsifier":"Simulate independent uniform p-values under the global null, form groups by the paper's application thresholds (p > 0.7, 0.15 < p < 0.7, p < 0.15), run the plug-in adaptive sGBH at level 0.05, and average the false discovery proportion; if it materially exceeds 0.05, the data-dependent grouping breaks the super-uniformity assumption that Theorem 2 requires.","tokens_in":37176,"feed_emoji":"🎯","tokens_out":8386,"duration_ms":80363,"temperature":0.7,"pith_summary":"The paper introduces sGBH, a two-stage procedure for false discovery rate control when hypotheses are grouped and most groups contain no real effects. Rather than weighting every p-value, as grouped BH procedures do, sGBH first screens groups with a uniformity test, then weights and tests only the p-values in the groups it declares interesting. The paper proves that, when the group partition is fixed before the p-values are seen and groupwise null proportions are estimated by reciprocally conservative estimators, the FDR is bounded by the nominal level plus the probability of mis-selecting groups or mis-estimating proportions, and tends asymptotically to no more than the nominal level. The practical payoff is power: in simulations and a prostate cancer gene-expression analysis, sGBH reports more discoveries than standard grouped BH at the same nominal FDR.","feed_headline":"Test only promising groups and still control FDR","feed_subtitle":"Screening groups before weighting p-values preserves false discovery control and finds more signals in sparse settings.","key_machinery":"The load-bearing identity is the oracle sGBH weight $v_j = \\pi_{j0}(1-\\tilde\\pi_0)/(1-\\pi_{j0})$ applied only to p-values in selected groups, together with the identity $(1-\\tilde\\pi_0)m_S = (1-\\pi_0)m$, which makes the oracle sGBH and oracle GBH coincide under sparse configurations. The other central ingredient is a reciprocally conservative estimator of the proportion of true nulls, meaning $E[1/\\hat\\pi_{j0,k}] \\le 1/\\pi_{j0}$ when a true-null p-value is set to zero; this inequality is what lets the proof bound the FDR non-asymptotically. A KS uniformity test serves as the selection mechanism that recovers the set of interesting groups.","core_discovery":"A grouped, selectively weighted procedure can preserve FDR control while testing only a subset of hypotheses. Under a sparse configuration, where some groups contain no false nulls, sGBH estimates the set of interesting groups, weights p-values only inside those groups, and applies BH to the weighted p-values. Theorem 2 shows that if each groupwise null proportion estimator is non-increasing and reciprocally conservative, or consistent, and if the group-selection step consistently recovers the interesting set, then the plug-in adaptive sGBH is non-asymptotically conservative up to an explicit additive error, and asymptotically has FDR no larger than the nominal level. The paper also shows that the oracle versions of sGBH and GBH make exactly the same rejections under a nontrivial sparse configuration, so the practical power gain comes from how the adaptive versions estimate the interesting set and the null proportions.","pith_inferences":["A natural repair for the data-dependent grouping in the application is sample splitting: build the groups on one part of the data and test on the other, which would restore the theorem's requirement that the partition be fixed before seeing the tested p-values.","The reciprocal-conservativeness mechanism is not tied to groups specifically; any two-stage procedure that selects hypotheses and then reweights their p-values must account for the selection step, and the same inequality could be adapted to selection based on effect-size estimates rather than uniformity tests.","When groups are formed by p-value bins, true-null p-values in a bin follow truncated uniforms that are not super-uniform, so FDR inflation under the global null is expected; quantifying that inflation would provide a practical calibration for bin-based grouping."],"forward_implications":["With a fixed group partition, the plug-in adaptive sGBH controls FDR non-asymptotically up to an additive term that vanishes when the interesting set and the groupwise null proportions are consistently estimated.","Under a nontrivial sparse configuration, the oracle sGBH and oracle GBH reject exactly the same hypotheses, so any power advantage in practice must come from estimation of the interesting set and proportions rather than from the oracle weighting scheme.","The generic adaptive sGBH, using the weights of Nandi and Sarkar (2018), inherits non-asymptotic FDR control under independence and is nearly insensitive to its tuning parameter in simulations.","In the prostate cancer application, the plug-in adaptive sGBH reports 759 differentially expressed genes at FDR level 0.05, compared with 487 for the plug-in adaptive GBH.","As the total number of hypotheses grows, the power gap between the adaptive sGBH and the adaptive GBH shrinks, because the estimates of the interesting set and null proportions converge to their oracle values."],"supporting_citations":[{"why":"Defines the grouped GBH procedure and its oracle conservativeness, which sGBH extends and subsumes.","marker":"Hu et al. (2010)"},{"why":"Supplies the non-asymptotic conservativeness strategy that Theorem 2 generalizes to selectively weighted groups.","marker":"Chen et al. (2017)"},{"why":"Provides the generic weights used by the second adaptive version of sGBH and a one-way GBH that is non-asymptotically conservative under independence.","marker":"Nandi and Sarkar (2018)"},{"why":"Establishes reciprocal conservativeness and examples of non-increasing reciprocally conservative estimators, the key condition in Theorem 2.","marker":"Blanchard and Roquain (2009)"},{"why":"Defines the PRDS property used to extend conservativeness of the oracle sGBH under dependence.","marker":"Benjamini and Yekutieli (2001)"},{"why":"The KS test is the uniformity test used to estimate the set of interesting groups in Lemma 1.","marker":"Kolmogorov (1933) and Smirnov (1939)"},{"why":"Provides the KS test power convergence used in the proof of Lemma 1 to show consistent estimation of S.","marker":"Massey (1950)"},{"why":"Supplies the consistent estimator of the proportion of nonzero normal means used for groupwise null proportions in simulations and asymptotics.","marker":"Jin (2008)"},{"why":"Extends Jin's estimator to principal correlation structure and connects PCS with PRDS, supporting the asymptotic version of Theorem 2.","marker":"Chen (2018)"}],"fun_headline_variants":["Selective weighting boosts power while controlling FDR","Group screening then weighted BH: FDR safe","sGBH: Focus p-values, keep FDR in check","Sparse groups? Weight only interesting ones, control FDR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The FDR guarantee assumes the group partition is fixed before the p-values are seen, but the paper's own application builds the groups by thresholding the same p-values used for testing, which makes true-null p-values in a group follow truncated uniforms that are not super-uniform.","fun_headline_variants_meta":{"raw":{"variants":["Selective weighting boosts power while controlling FDR","Group screening then weighted BH: FDR safe","sGBH: Focus p-values, keep FDR in check","Sparse groups? Weight only interesting ones, control FDR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1677,"prompt_tokens":882,"completion_tokens":795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":729}},"tokens_in":498,"tokens_out":795,"duration_ms":8075,"temperature":1.0,"reasoning_tokens":729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:17:54.249655+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate independent uniform p-values under the global null, form groups by the paper's application thresholds (p > 0.7, 0.15 < p < 0.7, p < 0.15), run the plug-in adaptive sGBH at level 0.05, and average the false discovery proportion; if it materially exceeds 0.05, the data-dependent grouping breaks the super-uniformity assumption that Theorem 2 requires.","supporting_citations":[],"review_version":1}