Pith. sign in

REVIEW 3 major objections 6 minor 29 references

Properties of adaptively weighted Fisher's method

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that the adaptively weighted Fisher's method assigns weight 1 exactly to studies with nonzero effects as sample sizes grow, and that it has the same asymptotic Bahadur slope as Fisher's method.

desk verdict The paper's new consistency theorem is false as stated when a nonzero-effect study has o(n) sample size; the intended result likely holds under a proportional-sample-size assumption that the authors forgot to state. read the letter →

arxiv 1908.00583 v1 pith:5PPRJFXH submitted 2019-08-01 math.ST stat.TH

Yusi Fang , Shaowu Tang , Zhiguang Huo , George C. Tseng , Yongseok Park This is my paper · ORCID
classification math.STstat.TH MSC 62F0362F0562F12
keywords adaptivelyweightedFisher'smethodcombiningp-valuesmeta-analysisconsistencyasymptoticBahaduroptimalityexactslopebinaryadaptiveweightsheterogeneity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Meta-analyses that combine p-values from several studies usually gain power but say nothing about which studies actually carry the signal. This paper studies the adaptively weighted Fisher (AW-Fisher) method, which assigns each study a binary weight of 0 or 1, and establishes two asymptotic facts: the estimated weights converge to the true set of studies with nonzero effects as sample sizes grow, and the test's exact Bahadur slope equals that of Fisher's method. A sympathetic reader cares because the weights are used in genomics to group genes by expression patterns across tissues; knowing the weights are consistent means that grouping is trustworthy in large samples, and knowing the slope matches Fisher's means the extra interpretability costs no asymptotic efficiency.

What carries the argument

The proof is carried by two asymptotic identities. First, if the p-value of study $k$ has exact slope $c_k(\theta)$, meaning $-2\log p_k/n_k \to c_k(\theta)$, then with proportional sample sizes $-2\log p_k/n \to \lambda_k c_k(\theta)$. Second, the chi distribution's survival function satisfies $\log(1-F_{\chi_m}(x)) \sim -x^2/2$ as $x \to \infty$, so the AW-Fisher loss $L(T(w;P))$ behaves exponentially with rate $\sum_k w_k \lambda_k c_k(\theta)$. Comparing the tail ratio of the true subset with any other subset exposes an exponential factor $\exp(n/2 \sum_j c_j)$ when real studies are dropped, and a factor $O(1/n^{\ell'})$ when null studies are included; these factors drive the probability of any wrong weight vector to zero. The same tail comparison sandwiches the AW-Fisher p-value between $L_{obs}$ and $(2^K - 1)L_{obs}$, transferring the consistency into equality of exact slopes.

What would settle it

Take $K=2$, put a nonzero effect in study 1 only, and let $n_1$ grow like $\log n_2$ so that $n_1/n_2 \to 0$; this regime is excluded by the theorem, and if simulation shows $P(\hat{w}_1 = 1)$ does not approach 1, then the proportional-growth assumption is not merely technical but necessary. Conversely, under equal effects one can estimate $-2\log P(S > s_{obs})/n$ by simulation and check whether it approaches $\sum_k \lambda_k c_k(\theta)$, as Theorem 2 requires.

Watch

Extended reading notes

Core claim

Let $p_k$ be the p-value from study $k$ and let $T(w;P) = -2\sum_k w_k \log p_k$ be Fisher's statistic on a weighted subset. The AW-Fisher statistic is $s(P) = -\log(\min_w L(T(w;P)))$, where the minimum is over all nonzero binary weight vectors and $L$ is the chi-square survival probability; the selected weight vector $\hat{w}$ minimizes $L$. Theorem 1 states that if study sample sizes grow proportionally, $n_k/n \to \lambda_k$, then $\hat{w} \to w^*$ with probability tending to 1, where $w^*_k = 1$ exactly when $\theta_k \neq 0$. Theorem 2 states that, when all nonzero effects take a common value $\theta \neq 0$, the exact slope satisfies $c_{AW}(\theta) = c_{Fisher}(\theta) = \sum_k \lambda_k c_k(\theta)$, so AW-Fisher is asymptotically Bahadur optimal alongside Fisher's method.

Load-bearing premise

Every study's sample size must grow at the same rate as the average sample size, so $n_k/n$ has a positive limit; otherwise a study with negligible relative sample size cannot be reliably detected by the weight estimator.

Editorial extensions

If this is right

  • As all study sample sizes grow with fixed proportions, AW-Fisher's selected weight vector converges to the indicator of nonzero-effect studies, so the gene-by-tissue pattern categories read off the weights are asymptotically correct.
  • When all true effects share a common nonzero value, AW-Fisher has the same exact Bahadur slope as Fisher's method, so it carries no asymptotic Bahadur efficiency penalty relative to Fisher's classic test.
  • Weight errors are not symmetric: a true study is dropped with probability decaying like $n \exp(-n c_k \lambda_k /2)$, while a null study is kept with probability only $O(1/n)$, so misclassification of real signals disappears much faster than spurious inclusions.
  • Because $\hat{w}$ is consistent and the AW-Fisher p-value lies between $L_{obs}$ and a constant multiple of $L_{obs}$, the test's tail probability and the chi-square tail yield the same slope, which is the direct route to asymptotic Bahadur optimality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the proportional-sample-size condition fails, for example one study's sample size is $o(n)$ while another grows linearly, the exponential separation in the consistency proof becomes polynomial and weight consistency may fail; a testable extension is to identify the largest unbalanced growth under which $\hat{w}$ still recovers $w^*$.
  • The tail-ratio comparison suggests a finite-sample formula for misclassification probabilities, so fitting that formula to simulated or real data could give practical uncertainty estimates for learned weights.
  • Because asymptotic Bahadur optimality is proven under a common nonzero effect $\theta_k \equiv \theta$, a natural follow-up is to derive the exact slope of AW-Fisher under mixed directions or heterogeneous magnitudes, where its subset selection may diverge from Fisher's slope.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript studies the adaptively weighted Fisher (AW-Fisher) method for combining p-values from independent studies, where binary weights are used to indicate which studies contribute to the overall signal. The paper claims two main results: Theorem 1, that the estimated AW weight vector converges to the true vector marking exactly the studies with nonzero effects; and Theorem 2, that the AW-Fisher test has the same exact Bahadur slope as Fisher's method and is therefore asymptotically Bahadur optimal. The proofs use exact-slope asymptotics and chi-square tail expansions, and simulations are provided for the convergence rates of the weight estimates.

Significance. If the main results were correct, they would substantially strengthen the theoretical justification of AW-Fisher, a method already used in genomic meta-analysis because its estimated weights provide biological information about study-level heterogeneity. Equally important, the claimed asymptotic Bahadur optimality would place AW-Fisher on par with Fisher's method in terms of asymptotic efficiency while preserving the interpretability advantage. The paper also addresses a practical need, and the simulation setup is natural. However, the central consistency theorem is false as stated, and the proof of the Bahadur optimality theorem contains serious algebraic and logical gaps. These issues need to be resolved before the paper can be considered acceptable.

major comments (3)
  1. [Section 3, proof of Theorem 1] Theorem 1 is false as stated because the assumption lim n_k/n = lambda_k does not require lambda_k > 0. For a study with nonzero effect but lambda_k = 0, the claimed weight consistency can fail. A concrete counterexample is K = 2, theta_1 = theta_2 = 0.5, n_1 = n - floor(log n), n_2 = floor(log n). Here lambda_1 = 1 and lambda_2 = 0, yet theta_2 is nonzero. Since -2 log p_2 is of order (theta_2^2/4) log n, the tail-ratio L({1,0})/L({1,1}) behaves like n^{theta_2^2/8 - 1} = n^{-0.96875}, which tends to 0, so AW-Fisher asymptotically selects the weight vector (1,0) rather than w* = (1,1). Thus the statement 'w_hat -> w*' requires an additional condition such as lambda_k > 0 for every k with theta_k nonzero, or at least n_k / log n -> infinity for such studies. This is a load-bearing error because it invalidates the stated consistency result.
  2. [Section 3, proof of Theorem 1] The proof of Theorem 1 contains an algebraic error in the tail-ratio expansion. The authors replace -2 sum_{j=1}^k log p_j by n C_k with C_k = sum_{j=1}^k lambda_j c_j(theta), but in the exponential factor they write exp{n/2 (C_{i+ell} - C_i)} and then replace this by exp{n/2 sum_{j=1}^ell c_{i+j}}, dropping the lambda_j factors. The correct exponent should be exp{n/2 sum_{j=1}^ell lambda_{i+j} c_{i+j}}. This matters because if lambda_{i+ell} = 0 for a nonzero-effect study, the exponent vanishes and the polynomial factor (n/2)^{-ell} makes the ratio tend to 0, not infinity. The proof's conclusion that the ratio diverges is therefore not supported by its own expansion, and the claimed convergence rate O(n^ell exp(-n/2 sum c_{i+j})) should be O(n^ell exp(-n/2 sum lambda_{i+j} c_{i+j})) when lambda factors are included.
  3. [Section 4, proof of Theorem 2] The proof of Theorem 2 has a serious algebraic mistake in the displayed chain after the Bonferroni bound. The text writes 'lim -2/n log(P(S > s_obs)) > lim -2/n {L_obs + log(2K-1)} = sum w*_i lambda_i c_i(theta)', but the logarithm must apply to L_obs: the correct inequality is -2/n log(P(S > s_obs)) >= -2/n(log L_obs + log(2K-1)). As written, the expression -2/n{L_obs + log(2K-1)} has the wrong scale and does not converge to the claimed slope. This is not a typographical nuisance; it breaks the derivation of the exact slope. In addition, the lower-bound part invokes the random w_hat where a fixed subset w* is required: because w_hat is a data-dependent minimizer, one cannot simply replace it by w* inside the limit without an independent argument. The consistency of the weights, even if proven, does not directly justify the interchange of the random weight sequence with the deterministic limit in the log-tail expression. Thus the theorem's conclusion c_AW = c_Fisher is not established on the basis given in the manuscript.
minor comments (6)
  1. [Section 1] The phrase 'Asymptotic Barhadur optimality' contains a misspelling of Bahadur; the correct spelling is used elsewhere in the paper.
  2. [Section 2 and Section 5] There are repeated typos such as 'statisfies' for 'satisfies' and 'effective size' for 'effect size'.
  3. [Section 5] The fitted parameter values in the first simulation are missing: the text reads 'The estimates are â = and b̂ = .' The authors should provide the actual numerical values.
  4. [Figure 2 caption] The caption says 'Comparing accuracy of the new approach and permutation approach to obtain the AW p-values,' but the figure actually displays convergence rates of the weight estimates. The caption should be corrected to match the displayed content.
  5. [Header, page 14] The submission header states 'Received October 2007. Revised February 2008. Accepted March 2008.' These dates are inconsistent with the arXiv submission date of August 2019 and should be corrected.
  6. [Lemma 2] The notation is confusing: F_chi_k(x) is used as the CDF of the chi distribution, since x appears inside log(1 - F_chi_k(x)) ~ -x^2/2, but the sentence says 'chi^2_k follows chi-square distribution.' Please clarify that the variable x is the square root of a chi-square variable.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the AW-Fisher consistency and ABO proofs are built from exact-slope definitions and external Bahadur/Littell-Folks results; the Li et al. self-citation supplies the method, not the proof.

full rationale

The paper's two main theorems are not circular. Theorem 1 derives weight consistency from the definition of exact slope (-2/n_k log p_k -> c_k(θ)) and the chi-square tail expansion attributed to Bahadur et al. (1960); the target w* is defined by effect sizes θ_k, with the standard exact-slope property c_k>0 iff θ_k≠0 stated as background, not fitted. Theorem 2 computes the exact slope of the AW statistic by sandwiching P(S>s_obs) between L_obs and (2^K-1)L_obs and comparing with Littell-Folks's externally established exact slope of Fisher's method; no parameter estimated from data enters either proof. The only self-citation is Li et al. (2011), co-authored by Tseng, which is the origin of the AW-Fisher statistic itself; it is used to define the object of study and to recall admissibility, not to justify the new asymptotic claims, so it is not load-bearing. The cited Huo et al. (2017) concerns computation, not the theorems. The consistency proof does contain a substantive mathematical gap: it assumes lim n_k/n=λ_k but never requires λ_k>0, and the displayed tail-ratio expansion silently replaces Σλ_j c_j with Σ c_j, so for a nonzero-effect study with n_k=o(n) the exponential term collapses and the claimed divergence reverses. That is a correctness flaw in the stated theorem, not a circular reduction of the conclusion to the inputs. Accordingly, no derivation step is equivalent to its own input by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The theoretical claims use no free parameters; the only fitted quantities are the simulation curve coefficients a and b, whose values the manuscript leaves blank. The central theorems rest on the exact-slope model, proportional sample sizes, independence, and two external results, Bahadur's tail expansion and Littell-Folks's ABO theorem. No new theoretical entities are introduced; the 0/1 weights are defined by Li et al. (2011).

free parameters (2)
  • a = not reported (blank in manuscript)
    Coefficient in fitted curve P(w=0|w*=1)/n = a exp(-bn) in simulation 1 (Section 5); used only to illustrate convergence rate, not in the theorem.
  • b = not reported (blank in manuscript)
    Coefficient in both fitted curves in Section 5; used only to illustrate convergence rate, not in the theorem.
assumptions (5)
  • domain assumption Exact slope condition: for each study k, -2/n_k log p_k -> c_k(theta) as n_k -> infinity, with c_k(theta) > 0 iff theta_k != 0.
    Invoked at the start of Section 3 to define the consistency proof; if p-values do not have exact slopes, the chi-square tail ratio arguments fail.
  • domain assumption Proportional sample sizes: lim n -> infinity n_k/n = lambda_k for each study k.
    Stated in Section 3 immediately before Theorem 1; the exponential separation in Theorem 1 requires positive lambda_k.
  • domain assumption Independence of studies and uniform null p-values.
    Section 2 sets up K independent studies; the null distribution T(w) ~ chi-square and L(w) uniform under H0 are used throughout Section 4.
  • standard math Chi distribution tail expansion (Lemma 2) from Bahadur et al. (1960): log(1 - F_chi_k(x)) = -x^2/2 (1 + o(1)).
    Used in Theorem 2 to convert Lobs into an exact slope; external result attributed to Bahadur et al.
  • standard math Littell-Folks theorem: Fisher's method is asymptotically Bahadur optimal among p-value combination methods.
    External benchmark in Section 4; AW's ABO is concluded by equality with Fisher's exact slope.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Properties of adaptively weighted Fisher's method." pith.science (2026). https://pith.science/paper/5PPRJFXH

@misc{pith2026190800583,
  author       = {Pith},
  title        = {Pith review of: Properties of adaptively weighted Fisher's method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5PPRJFXH}},
  note         = {Machine review of arXiv:1908.00583}
}
abstract

Meta-analysis is a statistical method to combine results from multiple clinical or genomic studies with the same or similar research problems. It has been widely use to increase statistical power in finding clinical or genomic differences among different groups. One major category of meta-analysis is combining p-values from independent studies and the Fisher's method is one of the most commonly used statistical methods. However, due to heterogeneity of studies, particularly in the field of genomic research, with thousands of features such as genes to assess, researches often desire to discover this heterogeneous information when identify differentially expressed genomic features. To address this problem, \citet{Li2011adaptively} proposed very interesting statistical method, adaptively weighted (AW) Fisher's method, where binary weights, $0$ or $1$, are assigned to the studies for each feature to distinguish potential zero or none-zero effect sizes. \citet{Li2011adaptively} has shown some good properties of AW fisher's method such as the admissibility. In this paper, we further explore some asymptotic properties of AW-Fisher's method including consistency of the adaptive weights and the asymptotic Bahadur optimality of the test.

Figures

Figures reproduced from arXiv: 1908.00583 by the authors.

Figure 1
Figure 1. Heatmaps of gene expressions for DE genes identified by Fisher’s and AW￾Fisher’s methods in the mouse energy metabolism datasets. (A) shows heatmap of gene expressions for DE genes identified by Fisher’s method with false discovery rate 1% (555 genes). For each of the three tissues (Brown fat, Liver and Heart), the group labels are on top of the heatmap, with black color represents wild type (WT) and gray color repr… view at source ↗
Figure 2
Figure 2. Comparing accuracy of the new approach and permutation approach to obtain the AW p-values. The scattered plots are p-values from the two methods against the closed￾form solution. ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● 200 400 600 800 1000 0.00 0.02 0.04 0.06 0.08 0.10 prob(w=0 | theta=1) n pr… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 28 canonical work pages

  1. [1]

    Bahadur, R. R. (1967). Rates of convergence of estimates and test statistics. Annals of Mathematical Statistics 38, 303--324

    work page 1967

  2. [2]

    Bahadur, R. R. et al. (1960). Stochastic comparison of tests. Annals of Mathematical Statistics 31, 276--295

    work page 1960

  3. [3]

    B., Wheeler, W

    Bhattacharjee, S., Rajaraman, P., Jacobs, K. B., Wheeler, W. A., Melin, B. S., Hartge, P., Yeager, M., Chung, C. C., Chanock, S. J., Chatterjee, N., et al. (2012). A subset-based approach improves power and interpretation for the combined analysis of genetic association studies of heterogeneous traits. The American Journal of Human Genetics 90, 821--835

    work page 2012

  4. [4]

    Fisher, R. A. (1925). Statistical methods for research workers . Genesis Publishing Pvt Ltd

    work page 1925

  5. [5]

    Flutre, T., Wen, X., Pritchard, J., and Stephens, M. (2013). A statistical framework for joint eqtl analysis in multiple tissues. PLoS Genet 9, e1003486

    work page 2013

  6. [6]

    J., Penny, W

    Friston, K. J., Penny, W. D., and Glaser, D. E. (2005). Conjunction revisited. Neuroimage 25, 661--667

    work page 2005

  7. [7]

    and Eskin, E

    Han, B. and Eskin, E. (2012). Interpreting meta-analyses of genome-wide association studies. PLoS Genet 8, e1002555

    work page 2012

  8. [8]

    Lancaster, H. (1961). The combination of probabilities: an application of orthonormal functions. Australian Journal of Statistics 3, 20--33

    work page 1961

Show all 29 references
  1. [9]

    C., et al

    Li, J., Tseng, G. C., et al. (2011). An adaptively weighted statistic for detecting differential gene expression when combining multiple transcriptomic studies. The Annals of Applied Statistics 5, 994--1019

    2011

  2. [10]

    and Ghosh, D

    Li, Y. and Ghosh, D. (2014). Meta-analysis based on weighted ordered p-values for genomic data with heterogeneity. BMC bioinformatics 15, 226

    2014

  3. [11]

    Littell, R. C. and Folks, J. L. (1971). Asymptotic optimality of fisher's method of combining independent tests. Journal of the American Statistical Association 66, 802--806

    1971

  4. [12]

    Littell, R. C. and Folks, J. L. (1973). Asymptotic optimality of fisher's method of combining independent tests. Journal of the American Statistical Association 68, 193--194

    1973

  5. [13]

    and Tseng, G

    Song, C. and Tseng, G. C. (2014). Hypothesis setting and order statistic for robust genomic meta-analysis. The annals of applied statistics 8, 777

    2014

  6. [14]

    A., Suchman, E

    Stouffer, S. A., Suchman, E. A., DeVinney, L. C., Star, S. A., and Williams Jr, R. M. (1949). The American soldier, Vol. I : adjustment during army life.(Studies in social psychology in World War II ) . Princeton Univiversity Press

    1949

  7. [15]

    Tippett, L. H. C. et al. (1931). The methods of statistics. The methods of statistics

    1931

  8. [16]

    Wilkinson, B. (1951). A statistical consideration in psychological research. Psychological Bulletin 48, 156

    1951

  9. [17]

    Birnbaum A. (1954). Combining independent tests of significance. Journal of the American Statistical Association, 49: 559-574

    1954

  10. [18]

    and Hedges L.V

    Cooper H.M. and Hedges L.V. (Eds.) (1994). The handbook of research synthesis. New York: Russell Sage Foundation

    1994

  11. [19]

    Folks J.L. (1984). Combination of independent tests. In Handbook of statistics 4. Nonparametric methods, P.R. Krishnaiah and P.K. Sen (eds):New York, North-Holland

    1984

  12. [20]

    and Olkin I

    Hedges L.V. and Olkin I. (1985). Statistical methods for meta-analysis. Academic Press Inc.: Orlando, Florida

    1985

  13. [21]

    and Vevea J.L

    Hedges L.V. and Vevea J.L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3:486-504

    1998

  14. [22]

    Hedges L.V. (2007). Meta-analysis. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp.919-953), New York: Russell Sage Foundation

    2007

  15. [23]

    and Schmidt F.L

    Hunter J.E. and Schmidt F.L. (2000). Fixed effects vs. random effects meta-analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8:275-292

    2000

  16. [24]

    and Saner H

    Olkin I. and Saner H. (2001). Approximations for trimmed Fisher procedures in research synthesis. Statist. Methods Med. Res., 10: 267-276

    2001

  17. [25]

    Owen A.B. (2009). Karl pearson's meta-analysis revisited. Annals of Statistics, 37: 3867-3892

    2009

  18. [26]

    Roy S.N. (1953). On a heuristic method of test construction and its use in multivariate analysis. The Annals of Mathematical Statistics, 24(2): 220-238

    1953

  19. [27]

    and Tseng G.C

    Song C. and Tseng G.C. (2014). Order statistic for robust genomic meta-analysis. Annals of Applied Statistics. Accepted

    2014

  20. [28]

    Stochastic comparison of tests

    Bahadur,R.R (1960). Stochastic comparison of tests. Annals of Mathematical Statistics,31(2): 276--295

    1960

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.