Efficient and powerful equivalency test on combined mean and variance with application to diagnostic device comparison studies

Baolin Wu; Theodore Lystig; Yun Bai; Zengri Wang

REVIEW 2 major objections 4 minor 18 references

Efficient and powerful equivalency test on combined mean and variance with application to diagnostic device comparison studies

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper develops a generalized pivotal test that claims exact parameter-free inference for the root mean square $\rho$ in paired repeated-measures device comparisons, outperforming large-sample approximations.

desk verdict Promising generalized pivotal test for RMS equivalence, but the Qb construction can fail to exist and needs referee attention. read the letter →

arxiv 1908.07979 v1 pith:5JKHVC4V submitted 2019-08-21 stat.ME stat.AP

Yun Bai , Zengri Wang , Theodore Lystig , Baolin Wu This is my paper · ORCID

classification stat.MEstat.AP MSC 62F0362J1062P10

keywords equivalencetestinggeneralizedpivotalquantityrootmeansquarepairedrepeatedmeasureslinearmixedeffectsmodelpulseoximetrytypeIerrorconfidenceinterval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper develops a test for whether two measurement methods agree, judged by the root mean square $\rho$ of their paired differences, in studies with repeated measures per subject. It claims that $\$rho^{2}$$ can be decomposed into three generalized pivotal quantities whose joint distribution contains no unknown parameters once the observed summary statistics are fixed, so the tail probability $\Pr(Q \ge \$rho_0^{2}$)$ supplies a valid p-value for the equivalence hypothesis $H_0: \rho \ge \rho_0$. Existing approaches rely on large-sample normal approximations and are conservative, especially at $\$\alpha$=0.01$; simulations show the new test controls type I error and has higher power. Applied to a pulse oximetry comparison, it gives a p-value of 0.006 and a narrower confidence interval than the Z-score test, supporting device equivalency.

What carries the argument

The central object is the generalized test statistic $Q = Q_w + Q_b + Q_\mu$. $Q_w = sse/(SSE/\$sigma_w^{2}$)$ has a scaled inverse chi-square distribution; $Q_b = h(\bar{y}, Q_w, SSR)$ is defined as the solution to the weighted between-subject sum-of-squares identity (2) for $\$sigma_b^{2}$$; $Q_\mu = [\tilde{y} - Z(\sum_i \tilde{W}_i)^{-1/2}]^2$ is the squared normal pivot for the mean using the same simulated variance components. The machinery works because, conditional on the observed summary statistics, the randomness in $Q$ comes only from $\$chi^{2}$_{N-n}$, $\$chi^{2}$_{n-1}$, and $N(0,1)$ variables, so the distribution of $Q$ is free of unknown parameters; and because $Q$ collapses to $\$rho^{2}$$ at the observed data, tail probabilities of $Q$ serve as p-values. A numerical step integrates out $Z$ analytically, reducing the tail probability to a noncentral chi-square calculation.

What would settle it

Simulate null data with small n (for example n=10 subjects with m=5 repeated measures), $\rho=\rho_0$, and between-subject variance large relative to within-subject variance, compute the generalized p-value $\Pr(Q \ge \$rho_0^{2}$)$ over many replicates, and check whether the empirical rejection rate at $\$\alpha$=0.05$ stays within binomial tolerance of 0.05. Also record the fraction of Monte Carlo draws of $(SSR, Q_w)$ for which the equation defining $Q_b$ has no positive real solution or yields multiple solutions; a non-negligible fraction would break the distributional claim.

Watch

Extended reading notes

Core claim

The paper claims that the root mean square parameter $\rho = \sqrt{\$mu^{2}$ + \$sigma_b^{2}$ + \$sigma_w^{2}$}$ in a one-way random-effects model can be tested by a generalized pivotal quantity $Q = Q_w + Q_b + Q_\mu$. $Q_w$ follows a scaled inverse chi-square distribution conditional on the observed within-subject sum of squares; $Q_b$ is obtained by solving equation (2) for $\$sigma_b^{2}$$ as a deterministic function of the observed sample means, the simulated $Q_w$, and a chi-square between-subject sum of squares; $Q_\mu$ is the squared normal pivot for the mean. The distribution of $Q$ depends only on known distributions once the observed summary statistics $(\bar{y}, sse)$ are fixed, and $Q$ reduces to $\$rho^{2}$$ at the observed data. Therefore the paper asserts that $\Pr(Q \ge \$rho_0^{2}$)$ is a valid generalized p-value for $H_0: \rho \ge \rho_0$, and that the percentiles of $Q$ give a generalized confidence interval for $\rho$.

Load-bearing premise

The load-bearing premise is that the mathematical relationship used to pin down the between-subject variance $\$sigma_b^{2}$$ has a unique solution for every simulated draw and that the resulting quantity $Q_b$ has a distribution free of unknown parameters; the paper assumes this without giving a proof or explicit algorithm.

Editorial extensions

If this is right

Equivalence testing of pulse oximeters against an FDA threshold can be performed at small to medium sample sizes with type I error near the nominal level, instead of conservative Z-score tests.
The generalized confidence interval for $\rho$ has coverage close to the nominal 90% and is narrower on average than intervals from the large-sample Z-score approach.
Because the method needs only the per-subject sample sizes, means, and the residual sum of squares, it can be applied to published summary statistics without raw data.
At the stringent $\alpha=0.01$ level, the proposed test preserves power in settings where the Z-score test nearly loses power, for example 49.3% versus 3.2% in the unbalanced n=16 scenario.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same three-part pivot decomposition could be adapted to test other composite parameters that are sums of a squared mean and variance components, such as the coefficient of variation, provided the solving step for the variance component remains tractable.
Editorial inference: the observed-data collapse $Q \to \rho^2$ suggests the generalized confidence interval could be inverted for sample-size planning of equivalence studies, a use the paper does not develop.
Editorial inference: because the paper does not prove uniqueness of the solution to equation (2), a numerical study of how often the solving step produces no root or multiple roots across parameter configurations would clarify the method's practical domain.
Editorial inference: the analytical integration over $Z$ implies the method's Monte Carlo error comes only from the chi-square draws, so the p-value could in principle be computed deterministically to arbitrary precision.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear X,

You should know about this paper because its core statistic Qb is not actually defined for all Monte Carlo draws. In Section 2.1 the authors solve eq. (2) for sigma_b^2 as a function of observed group means, Qw, and a simulated chi-square SSR, but as written the equation only has a nonnegative solution when SSR is no larger than the maximum L(0) = sum_i m_i(bar y_i - bar y_w)^2 / Qw. Simulated SSR can exceed L(0), and in small-sigma_b^2 regimes it will do so often. The paper gives no fallback, no algorithm, and no proof of existence; it simply states 'we can solve.' That is a real gap in the generalized inference claim, because the distribution of Q would not be parameter-free if a solution sometimes fails to exist.

That said, the idea is genuinely new and the paper does much right. Combining generalized pivotal quantities for the mean and variance components to test the root-mean-square parameter in a one-way random effects model is not in the cited literature, and the simulation study is extensive: 10^4 reps, unbalanced and balanced designs, multiple variance ratios, and the results consistently show better power than the Z-score test at small samples while keeping type I error below nominal. The oximetry example and the public R package are useful contributions. The conservative type I rates are a minor issue, not a fatal one, since conservatism is generally acceptable in equivalence testing.

The soft spot is the construction of Qb. I checked the stress-test note and it holds up: for fixed observed group means and simulated Qw, the weighted between-group sum of squares is strictly decreasing in sigma_b^2 and bounded above by its value at zero, so a solution exists only if the simulated SSR falls below that bound. The simulation configurations keep sigma_b^2 well away from zero (ratios 1/3, 1, and 3), so the failure region is never exercised. The authors need to state how the R package handles non-solvable draws — clipping, rejecting, re-drawing — and show that the resulting distribution of Qb is still correct, or modify the pivot so a solution always exists. Without that, the exactness claim in Section 2.2 is not established.

Who benefits: statisticians working on FDA submissions for pulse oximeters and similar paired device comparisons. It deserves a serious referee, but the referee should ask for a precise existence proof or a corrected algorithm, plus simulations near sigma_b^2 = 0. I would not desk-reject it. Recommend peer review with major revision.

Referee Report

2 major / 4 minor

Summary. The manuscript develops a generalized pivotal quantity test for the equivalency hypothesis H0: ρ ≥ ρ0 versus Ha: ρ < ρ0, where ρ = sqrt(μ² + σ_b² + σ_w²) is the root mean square of paired differences in a one-way random-effects model. The method constructs pivots Qw for σ_w², Qb for σ_b² (by inverting the between-group weighted sum-of-squares identity), and Qμ for μ², and uses Q = Qw + Qb + Qμ as the test statistic. The authors report simulation studies with 10^4 replications showing controlled type I error and improved power relative to a Z-score test, illustrate the method on a pulse oximetry equivalency study, and provide an R package.

Significance. If the pivot construction is rigorously valid, this paper fills a practically important gap: a small-sample test for the FDA-relevant RMS parameter in paired repeated-measures device comparison studies. The simulation evidence is extensive and the publicly available R package is a valuable contribution; the power advantage over the Z-score test, especially at α = 0.01, is substantial in the studied settings. However, the load-bearing definition of Qb has a gap that must be addressed before the claims of a parameter-free pivot distribution and exact generalized inference can be accepted.

major comments (2)

[§2.1, Eq. (2)–(3)] The existence of the function h used to define Qb is not established for all Monte Carlo draws. For fixed observed group means ȳ_i and a simulated Qw, the right-hand side of Eq. (2) is a strictly decreasing function of σ_b² on [0,∞) with supremum L(0) = Σ m_i(ȳ_i − ȳ_w)²/Qw; therefore a nonnegative solution exists only when the simulated SSR is no larger than L(0). Simulated SSR ∼ χ²_{n−1} can exceed L(0) with non-negligible probability, especially when σ_b² is near zero, so the non-solvable set is not measure zero. Since Qb is undefined on those draws, the statement in §2.1 that Qb has a parameter-free distribution and the generalized p-value construction in §2.2 are not justified. The authors must either prove the event has probability zero (which is not the case), or redefine the pivot so it is always well-defined, and they must specify the numerical algorithm used in the R package.
[§3.1–3.2, Tables 1–4 and S1–S6] The simulation grid never exercises the σ_b² near zero boundary where the existence failure above is most severe; the smallest σ_b² configuration used is σ_w²:σ_b² = 3:1, which keeps σ_b² at 25% of the total variance component. If the implementation silently handles non-solvable draws (for example by clipping Qb to zero or resampling SSR), the distribution of Q changes and the reported type I error and power results do not necessarily describe the method as defined in §2.1. The paper should report the frequency of non-solvable draws across configurations and validate any fallback rule.

minor comments (4)

[§3.2] The text says "Table S6 summarizes the power for the unbalanced study," but Table S6 reports balanced scenarios with n=30; the unbalanced-study power appears in Table 3. The cross-reference should be corrected.
[§2.4] The Z-score and Z-Wald tests are named but their construction is not fully specified; equation (8) gives Var(R), but the paper should state explicitly how the variance is estimated under the null for the score test and at the MLE for the Wald test.
[§2.3] The notation Sχ²_1(·|λ) is introduced without a clear definition of the argument order; the displayed formula for Pr(Q ≥ ρ0² | Qw,Qb) should define whether the first argument is the threshold or the noncentrality parameter.
[Appendix A] The appendix uses sse for the within-group sum of squares while the main text uses SSE for the random variable; the notation should be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generalized pivotal construction is self-contained and does not reduce to its own inputs.

full rationale

The paper's derivations in Sections 2.1 and 2.2 construct the test statistic Q = Qw + Qb + Qµ from standard parameter-free distributions: Qw via a scaled inverse chi-square from SSE, Qb via a stated deterministic function h of observed means, Qw and a chi-square SSR, and Qµ via a standard normal pivot. The claimed property that Q has a parameter-free distribution given the observed data, and that Q reduces to ρ² at the observed data, follows directly from these definitions rather than being assumed as an input. No fitted constant is renamed as a prediction, and no quantity is defined in terms of the target RMS parameter. Citations to the generalized inference literature (Tsui and Weerahandi, Weerahandi, Iyer et al.) are external standard references, not self-citations by the present authors, and they are not used to smuggle in an ansatz. The numerical studies benchmark the method against existing Z-score and Z-Wald tests, giving independent empirical support. The unresolved concern that equation (2) may not have a nonnegative solution for σ_b² for every simulated (Qw, SSR) draw is a soundness/existence gap in the construction, not a circular step: the claimed distribution-free property would fail or require a fallback, but the derivation is not equivalent to its own conclusion by construction. Therefore no circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The method rests on the one-way random effects ANOVA model, on the standard distributional results for unbalanced sums of squares, and on the unproved but essential solvability of equation (2). No new entities or fitted constants are introduced.

assumptions (3)

domain assumption The random effects u_i and error terms ε_ij are independent and normally distributed with zero means and variances σ_b^2 and σ_w^2 respectively.
The entire GPQ construction and distributional claims rest on this model; Section 2 opening.
ad hoc to paper Equation (2) can be solved uniquely for σ_b^2 as a function of (ȳ, σ_w^2, SSR).
The method requires that h in eq. (3) exists and is well-defined for all simulated values; the paper does not prove this.
standard math The distributional properties of generalized pivotal quantities (e.g., Qw ~ scaled inverse chi-square, SSR ~ chi-square n-1, Z ~ N(0,1)) hold as stated.
These follow from standard linear model theory if the normality assumptions hold and the parameters are known; used in Section 2.1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient and powerful equivalency test on combined mean and variance with application to diagnostic device comparison studies." pith.science (2026). https://pith.science/paper/5JKHVC4V

@misc{pith2026190807979,
  author       = {Pith},
  title        = {Pith review of: Efficient and powerful equivalency test on combined mean and variance with application to diagnostic device comparison studies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JKHVC4V}},
  note         = {Machine review of arXiv:1908.07979}
}

read the original abstract

In medical device comparison studies, equivalency test is commonly used to demonstrate two measurement methods agree up to a pre-specified performance goal based on the paired repeated measures. Such equivalency test often involves controlling the absolute differences that depend on both the mean and variance parameters, and poses some challenges for statistical analysis. For example, for the oximetry comparison study that motivates our research, FDA has clear guidelines approving an investigational pulse oximeter in comparison to a standard oximeter via testing the root mean squares (RMS), a composite measure of both mean and variance parameters. For the hypothesis testing of this composite measure, existing methods have been either exploratory or relying on the large-sample normal approximation with conservative and unsatisfactory performance. We develop a novel generalized pivotal test to rigorously and accurately test the system equivalency based on RMS. The proposed method has well-controlled type I error and favorable performance in our extensive numerical studies. When analyzing data from an oximetry comparison study, aiming to demonstrate performance equivalency between an FDA-cleared oximetry system and an investigational system, our proposed method resulted in a highly significant test result strongly supporting the system equivalency. We also provide efficient R programs for the proposed method in a publicly available R package. Considering that many practical equivalency studies of diagnostic devices are of small to medium sizes, our proposed method and software timely bridge an existing gap in the field.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

[1]

Bland, J. M. and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8, 135–160

work page 1999
[2]

Bland, J. M. and Altman, D. G. (2007). Agreement between meth ods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics 17, 571–582

work page 2007
[3]

K., Quiroz, J., and Iyer, H

Burdick, R. K., Quiroz, J., and Iyer, H. K. (2006). The presen t status of conﬁdence interval estimation for one-factor random models. Journal of Statistical Planning and Inference 136, 4307–4325

work page 2006
[4]

K., Wang, C

Iyer, H. K., Wang, C. M. J., and Mathew, T. (2004). Models and C onﬁdence Intervals for True V alues in Interlaboratory Trials. Journal of the American Statistical Association 99, 1060–1071. 12

work page 2004
[5]

Lee, Y ., Shao, J., and Chow, S.-C. (2004). Modiﬁed Large-Sam ple Conﬁdence Intervals for Linear Combinations of V ariance Components. Journal of the American Statistical Association 99, 467–478

work page 2004
[6]

Laird, N. M. and Ware, J. H. (1982). Random-Effects Models fo r Longitudinal Data. Biometrics 38, 963–974

work page 1982
[7]

Lin, L. I.-K. (1989). A Concordance Correlation Coefﬁcient to Evaluate Reproducibility. Biometrics 45, 255–268

work page 1989
[8]

Lin, L. I.-K. (1992). Assay V alidation Using the Concordanc e Correlation Coefﬁcient. Biometrics 48, 599–604

work page 1992

Show all 18 references

[9]

Ndikintum, N. K. and Rao, M. (2016). A Special Inference Prob lem in Repeated Measures Design Test of Statistical Hypothesis on Accuracy Root Mean Square Application to Pulse Oximetry Studies. Statistics in Biopharmaceutical Research 8, 60–76

2016
[10]

Park, D. J. and Burdick, R. K. (2003). Performance of Conﬁden ce Intervals in Regression Models with Unbalanced One-Fold Nested Error Structures. Communications in Statistics - Simulation and Computation 32, 717–732

2003
[11]

Park, D. J. and Burdick, R. K. (2004). Conﬁdence Intervals on Total V ariance in a Regression Model with an Unbalanced Onefold Nested Error Structure. Communications in Statistics - Theory and Methods 33, 2735–2743

2004
[12]

Pennello, G. A. (2002). Sample size for paired repeated meas ures designs of method comparison studies: Application to pulse oximetry. ASA Proceedings of the Joint Statistical Meetings pages 2671–2675

2002
[13]

Pennello, G. A. (2003). Comparing monitoring devices when a gold standard in unavailable: Application to pulse oximeters. ASA Proceedings of the Joint Statistical Meetings pages 3256–3263

2003
[14]

K., Graybill, F

Ting, N., Burdick, R. K., Graybill, F. A., Jeyaratnam, S., an d Lu, T.-F. C. (1990). Conﬁdence intervals on linear combinations of variance components that are unre stricted in sign. Journal of Statistical Computation and Simulation 35, 135–143. 13

1990
[15]

and Weerahandi, S

Tsui, K.-W. and Weerahandi, S. (1989). Generalized p-V alue s in Signiﬁcance Testing of Hypotheses in the Presence of Nuisance Parameters. Journal of the American Statistical Association 84, 602–607

1989
[16]

Weerahandi, S. (1993). Generalized Conﬁdence Intervals. Journal of the American Statistical Associa- tion 88, 899–905

1993
[17]

Weerahandi, S. (1995). Exact Statistical Methods for Data Analysis . Springer Series in Statistics. Springer-V erlag, New Y ork

1995
[18]

and Mathew, T

Zhou, L. and Mathew, T. (1994). Some Tests for V ariance Compo nents Using Generalized p V alues. Technometrics 36, 394–402. A Sufﬁcient statistics, likelihood calculation, and param eter estimation Denote Σ i = Cov(Yi) for i = 1,· · ·,n. We can write Σ i = σ 2 b 1mi 1T mi + σ...

1994

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Bland, J. M. and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8, 135–160

work page 1999

[2] [2]

Bland, J. M. and Altman, D. G. (2007). Agreement between meth ods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics 17, 571–582

work page 2007

[3] [3]

K., Quiroz, J., and Iyer, H

Burdick, R. K., Quiroz, J., and Iyer, H. K. (2006). The presen t status of conﬁdence interval estimation for one-factor random models. Journal of Statistical Planning and Inference 136, 4307–4325

work page 2006

[4] [4]

K., Wang, C

Iyer, H. K., Wang, C. M. J., and Mathew, T. (2004). Models and C onﬁdence Intervals for True V alues in Interlaboratory Trials. Journal of the American Statistical Association 99, 1060–1071. 12

work page 2004

[5] [5]

Lee, Y ., Shao, J., and Chow, S.-C. (2004). Modiﬁed Large-Sam ple Conﬁdence Intervals for Linear Combinations of V ariance Components. Journal of the American Statistical Association 99, 467–478

work page 2004

[6] [6]

Laird, N. M. and Ware, J. H. (1982). Random-Effects Models fo r Longitudinal Data. Biometrics 38, 963–974

work page 1982

[7] [7]

Lin, L. I.-K. (1989). A Concordance Correlation Coefﬁcient to Evaluate Reproducibility. Biometrics 45, 255–268

work page 1989

[8] [8]

Lin, L. I.-K. (1992). Assay V alidation Using the Concordanc e Correlation Coefﬁcient. Biometrics 48, 599–604

work page 1992