REVIEW 4 major objections 4 minor 2 references
Tissue-specific predictive performance: A unified estimation and inference framework for multi-category screening tests
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Under case-control sampling, unbiased estimates and Wald confidence intervals are derived for cancer-specific intrinsic accuracy, tissue-of-origin-specific predictive value, and the marginal test-readout distribution of multi-category…
desk verdict Useful unified framework for per-cancer-type MCED metrics, but the P(D) prior is load-bearing and needs formal sensitivity treatment before the predictive-value claims hold up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The unifying device is the compound random-variable estimator for intrinsic accuracy. With the total number of cases $N_1$ fixed but the realized number $n_{j\cdot}$ of cases in disease state $j$ random under multinomial sampling, the estimator $\hat A_{jk}=\tilde A_{jk}/P(n_{j\cdot}>0)$ is unbiased for $P(\mathrm{Test}=k\mid D_j)$, where $\tilde A_{jk}$ is the observed proportion of readout $k$ among the $n_{j\cdot}$ cases and the 0/0 case is defined as 0. Its variance comes from conditioning on whether disease state $j$ appears at all and on the reciprocal of the realized case count; the sample version substitutes $\hat p_j=n_{j\cdot}/N_1$ into the binomial law of $n_{j\cdot}$. These intrinsic-accuracy estimates feed, through Bayes rule with the external $P(D)$, into logit-transformed delta-method formulas for positive and negative predictive value and for marginal readout probabilities, with a covariance matrix that includes multinomial covariances among case-type proportions. A sparse-data adjustment adds $1/2$ to each control count when expected false-positive counts are below about five, stabilizing the predictive-value and marginal inferences.
What would settle it
Run a prospective screening cohort in the intended-use population with complete follow-up diagnosis, record the empirical tissue-of-origin-specific positive predictive value directly for each readout, and compare it with the paper's case-control estimate built from registry annual incidence $P(D)$; if the direct values differ from the framework's estimates by more than the reported confidence intervals for several readouts, the fixed external-incidence assumption is the part that fails.
Extended reading notes
Core claim
The central claim is that in a case-control study of $J$ mutually exclusive disease states against $K+1$ test-readout categories, every tissue-specific performance metric of clinical interest can be estimated unbiasedly with valid asymptotic confidence intervals, provided one external quantity—the disease incidence $P(D)$—is supplied. The authors express $P(D_j)=P(D_j\mid D)P(D)$, obtain $P(D_j\mid D)$ either from the case sample or from a registry, and propagate sampling uncertainty through logit-transformed delta-method variance formulas. For intrinsic accuracy $A_{jk}=P(\mathrm{Test}=k\mid D_j)$, they build estimators from compound random variables that account for the random number of cases in each disease state and show the estimators are unbiased. For predictive values and the marginal readout distribution, these estimates are combined with control false-positive rates and the fixed incidence value. The framework includes a sparse-count adjustment for very high specificity settings, stage-stratified decomposition of predictive value, and is supported by simulations and by re-analysis of a published MCED case-control dataset that yields per-cancer false-negative, intrinsic-accuracy, and per-readout predictive-value tables.
Load-bearing premise
The load-bearing premise is that the externally supplied disease incidence per person-year can stand in for the probability of disease in Bayes rule, even though in a screening population the pretest probability of detectable cancer may be several times larger than annual incidence.
Editorial extensions
If this is right
- Case-control data alone yield unbiased cancer-specific intrinsic accuracy with valid confidence intervals, so investigators can quantify per-cancer detection without waiting for prospective follow-up.
- Given an external or posited $P(D)$, tissue-of-origin-specific positive predictive value and the marginal readout distribution become estimable with confidence intervals, replacing aggregate PPV and 'TOO accuracy'.
- The sparse-count correction (adding $1/2$ to each control cell) restores near-nominal coverage and reduces bias when expected false-positive counts per readout are below five, the common high-specificity screening regime.
- Stage-stratified predictive value decomposes an overall readout PVP into early- and late-stage parts, so a favorable aggregate number can be checked for whether it is driven by late-stage detection.
- Re-analysis of the published MCED dataset produces per-cancer false-negative and intrinsic-accuracy estimates and per-readout predictive values with confidence intervals, showing that the reported aggregate 'TOO accuracy' and crude sensitivity are not clinically actionable per patient.
Reading between the lines
- Because every predictive-value and marginal estimate is monotone in the input $P(D)$, the framework could formally propagate uncertainty in incidence—for example a registry confidence interval or prior—rather than relying on a fixed point plus sensitivity analysis; that would widen intervals in proportion to incidence uncertainty.
- The $1/2$-count sparse-data correction is applied to control false-positive counts, but rare cancer types with few cases would similarly destabilize intrinsic-accuracy estimates, so a symmetric smoothing of case-side counts is a natural extension the paper leaves implicit.
- The readout categories need not be cancer sites: any mutually exclusive diagnostic labels, including AI classifier bins or hybrid assay/digital outputs, satisfy the same Bayes-rule machinery, so the method transfers to other multi-category screening or triage problems.
- If the intended-use setting is interval screening, replacing annual incidence with the pretest probability of detectable cancer over the screening round would likely raise every PVP; a prospective pilot's observed positive rate could calibrate that probability and test the sensitivity of the framework's estimates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a unified framework for estimating and drawing inference on cancer-type-specific accuracy metrics for multi-category screening tests (e.g., MCED tests) under case-control sampling. The authors define intrinsic accuracy, tissue-of-origin-specific positive predictive value, predictive value negative, and the marginal test readout distribution, and provide delta-method-based logit-scale Wald confidence intervals. They also propose a sparse-data correction for control counts, discuss stratified analyses, evaluate the methods in simulations, and apply them to a published MCED dataset.
Significance. If the identification assumptions are met, the paper fills a real gap: existing MCED evaluations typically report aggregate or crude sensitivity measures that do not provide the readout-specific predictive values needed for clinical decision-making. The analytical formulas are transparent, the simulation study is fairly extensive, and the data application illustrates the practical relevance. The main weakness is that the entire predictive-value framework rests on a single external input P(D), which the paper conflates with annual incidence; the validity of the reported intervals is therefore conditional on a quantity that is not identified by the case-control design. This issue is fixable but load-bearing.
major comments (4)
- [Section 2] The manuscript defines P(D) as "the expected number of cases per 1 person-year of follow-up" and then immediately uses 1-P(D)=P(D^c) in Bayes' rule. A per-person-year incidence rate is not a probability of disease at the time of testing; the correct Bayes prior for a screening test is the pretest probability of detectable cancer. Because Sections 4.2 and 4.3 express every PVP(k), PVN(k), and P(T=k) as a monotone function of P(D), the use of annual incidence systematically biases all predictive-value and marginal-distribution estimates. The simulation study in Section 6 cannot reveal this bias because the data are generated at the same P(D) that is later plugged into the estimator. Please reframe P(D) as a prevalence-type quantity and discuss how it should be sourced and how its uncertainty should be propagated.
- [Section 4.1] The derivation correctly shows that A_tilde_jk = n_jk/n_j# has expectation P(T_k|D_j) P(n_j#>0) and then defines the unbiased estimator A_hat_jk = A_tilde_jk / P(n_j#>0). However, the logit-based confidence interval is constructed from L(A_tilde_jk) = logit(n_jk/n_j#), not from logit(A_hat_jk). The variance formula is also for L(A_tilde_jk). Consequently, the interval targets logit(P(T_k|D_j)) only when P(n_j#>0) is essentially 1, and it is undefined when n_j#=0. The simulation settings all have large P(n_j#>0), so this inconsistency is not detected. Please either construct the interval from logit(A_hat_jk) and derive its variance, or explicitly justify why the bias in the logit transform is negligible in the intended applications.
- [Appendix A3] The covariance matrix V(phi_hat - phi) used for the predictive-value intervals relies on two assumptions stated without proof: E[Y_i^{(jk)} | Y_l^{(lk)}, n_j#, n_l#] = E(Y_i^{(jk)}) and P(n_j#>0 | n_l#>0) = P(n_j#>0). These assumptions are not self-evident because disease states are mutually exclusive and the case-type counts are negatively correlated under multinomial sampling. Since the variance formulas for U(phi_hat) and W(phi_hat) depend critically on these assumptions, they need justification or a sensitivity analysis showing that mild violations do not affect coverage.
- [Section 7] The empirical analysis is based on reconstructed counts for stomach and gallbladder cancer that are not present in the cited Liu et al. table: the authors assume 5 stomach cases with 1 false negative and 1 gallbladder case with 0 false negatives. The resulting Table 9 and Table 10 estimates are therefore not a direct analysis of the published data. This limitation should be stated prominently, and the sensitivity of the reported intrinsic accuracy and PVP estimates to these two assumptions should be assessed.
minor comments (4)
- [Abstract and Section 4] The phrase "unbiased estimation" is used throughout, but the predictive-value estimators are ratios and their unbiasedness is only asymptotic under the delta method. Please use "consistent" or "asymptotically unbiased" where appropriate.
- [Section 4.5] The variance of the adjusted control proportion is multiplied by N_0^2/(N_0+0.5(K+1))^2, which is only approximately 1 for large N_0. For small control samples this factor is not negligible, so ignoring it makes the intervals conservative rather than exactly calibrated; this should be stated explicitly.
- [Section 6] In the screening setting with N=500, the coverage for P(T_1) and P(T_2) is 93.4% and 93.1%, slightly below the nominal 95% level. This is not discussed; a brief comment on whether this reflects the sparsity or the logit approximation would be useful.
- [Figure 1] The scatter plot compares 1-PVP(k), which conditions on a positive test readout, with IA(k), which conditions on disease state. These are conditional probabilities in opposite directions, so the visual "cost-benefit" interpretation may be misleading without a clear statement of the different conditioning.
Circularity Check
No material circularity: intrinsic-accuracy estimators are derived from first principles, and predictive-value and marginal-distribution estimators are explicit plug-in functions of the disclosed external input P(D); the incidence-versus-probability status of P(D) is a correctness risk, not a circular step.
full rationale
Walking the derivation chain, no step reduces to its own inputs or to a fitted parameter renamed as a prediction. Section 4.1 derives the intrinsic-accuracy estimator A-hat_jk = A-tilde_jk / P(n_j+ > 0) as an unbiased compound-random-variable estimator, proving E[A-tilde_jk / P(n_j+ > 0)] = P(Test=k | D_j) and giving a delta-method logit-scale variance; this is first-principles estimation from the case-control sampling model and the multinomial case-type distribution, with the p-hat_j substitution explicitly flagged as a negligible-error approximation supported by the simulation. Sections 4.2 and 4.3 estimate PVP_k, PVN_k, and P(T=k) by substituting directly estimable binomial proportions (P(Test=k|D_j), P(Test=k|D_0)) and either registry-sourced or case-derived P(D_j|D) into closed-form Bayes-rule expressions that are functions of one external input, P(D). The paper discloses this input unambiguously: 'A value for overall disease incidence P(D) is assumed to be obtained from an external source... and is considered fixed for inferential purposes,' and Section 1 states the value 'can be varied in a sensitivity analysis if warranted.' Since P(D) is an external input rather than a quantity fitted to the target data, the predictive-value estimates do not constitute a fitted input called a prediction; they are ordinary plug-in estimators of explicitly stated functionals. The simulation study generates data at the posited P(D) and evaluates the estimators at the same P(D), which is the standard model-based way to verify unbiasedness and CI coverage under stated assumptions, not a circular confirmation. There are no self-citations in the reference list (SEER, Goodman, Liu et al., Mercaldo et al.), no uniqueness theorem imported from the authors' own prior work, and no ansatz smuggled in via citation; the sparse-cell half-count adjustment is Goodman's standard method, properly attributed. The remaining concern, which the skeptic reviewer correctly identifies, is that Section 2 defines P(D) as 'the expected number of cases per 1 person-year of follow-up' and asserts P(D) + P(D_c) = 1, thereby using an annual incidence rate as a Bayes prior probability, whereas the screening-relevant prior is the pretest probability of detectable cancer at the time of testing; mis-specifying P(D) biases every PVP, PVN, and P(T=k) estimate monotonically, and the Wald intervals only cover the sampling error conditional on the posited P(D).
Assumptions & free parameters
free parameters (1)
- Overall disease incidence P(D) =
Screen: 0.016; diagnostic: 0.07; SEER-derived in application
assumptions (4)
- domain assumption Case-control sample is a conditional random sample from the intended-use population; cases arise as a multinomial draw with probabilities P(D_j|D).
- domain assumption P(D) can be treated as a known fixed probability equal to disease incidence per person-year, and 1-P(D) is the probability of no disease.
- ad hoc to paper Conditional independence assumptions in Appendix A3: E[Y_i^{(jk)} | Y_l^{(lk)}, n_j, n_l] = E(Y_i^{(jk)}) and P(n_j>0 | n_l>0) = P(n_j>0).
- ad hoc to paper For the data application, reconstructed counts for stomach and gallbladder from Liu et al are accurate; assumptions: 5 stomach cases with 1 false negative, 1 gallbladder case with 0 false negatives.
Cite this review
Pith. "Pith review of Tissue-specific predictive performance: A unified estimation and inference framework for multi-category screening tests." pith.science (2026). https://pith.science/paper/65ORCD73
@misc{pith2026250521482,
author = {Pith},
title = {Pith review of: Tissue-specific predictive performance: A unified estimation and inference framework for multi-category screening tests},
year = {2026},
howpublished = {\url{https://pith.science/paper/65ORCD73}},
note = {Machine review of arXiv:2505.21482}
}
read the original abstract
Multi-Cancer Early Detection (MCED) testing with tissue localization aims to detect and identify multiple cancer types from a single blood sample. Such tests have the potential to aid clinical decisions and significantly improve health outcomes. Despite this promise, MCED testing has not yet achieved regulatory approval, reimbursement or broad clinical adoption. One major reason for this shortcoming is uncertainty about test performance resulting from the reporting of clinically obtuse metrics. Traditionally, MCED tests report aggregate measures of test performance, disregarding cancer type, that obscure biological variability and underlying differences in the test's behavior, limiting insight into true effectiveness. Clinically informative evaluation of an MCED test's performance requires metrics that are specific to cancer types. In the context of a case-control sampling design, this paper derives analytical methods that estimate cancer-specific intrinsic accuracy, tissue localization readout-specific predictive value and the marginal test classification distribution, each with corresponding confidence interval formulae. A simulation study is presented that evaluates performance of the proposed methodology and provides guidance for implementation. An application to a published MCED test dataset is given. These statistical approaches allow for estimation and inference for the pointed metric of an MCED test that allow its evaluation to support a potential role in early cancer detection. This framework enables more precise clinical decision-making, supports optimized trial designs across classical, digital, AI-driven, and hybrid stratified diagnostic screening platforms, and facilitates informed healthcare decisions by clinicians, policymakers, regulators, scientists, and patients.
Figures
Reference graph
Works this paper leans on
-
[5]
Stratified analyses using case-control sampling Analyses of subgroups of Table 2 arising from partitions defined by factors other than disease or test readout variables, e.g. demographic factors, proceed naturally using the corresponding methods above on the partitioned data. Stratified analysis of predictive-value and marginal test distribution metrics requ...
-
[8]
Discussion The concept of testing for the presence of multiple disease types (such as multiple cancers using MCED tests) in a single blood draw is very attractive. These tests require unbiased estimation techniques and valid inference procedures for relevant performance metrics on a per-disease type level to properly inform clinical utility. Evaluating MC...
work page 2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.