REVIEW 3 major objections 6 minor 22 references
Calibration of answer probabilities in verbal autopsies: Working Paper
T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Minimizing imputation error recovers the true cause-of-death probability table from unlabeled verbal autopsy questionnaires.
desk verdict Worth engaging — the imputation-accuracy idea is new and the paper is honest about its limits, but the abstract over-claims: uniqueness rests on a stipulated condition, not a derived one. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is the imputation cross-entropy I(qhat): delete one block of answers, compute the posterior over causes from the remaining answers using a VA algorithm and the candidate probbase, predict the deleted answer via F_k(pi,qhat) = sum_j pi_j qhat_jk, and score with cross-entropy. The expansion in Eq. (9) shows that under block-conditional independence (Assumption 1), the imputed probability equals the true conditional probability exactly when qhat = q. This identity carries the argument: it converts an unlabelled-data objective into a function whose minimum is the object of inference. Theorem 2's uniqueness rests on the strong condition that a VA algorithm returns the true poste
What would settle it
Construct or observe a dataset where the block-conditional-independence assumption provably holds but two distinct probbases give identical imputation scores for all inputs, contradicting Theorem 2's uniqueness; or, on a real labelled VA dataset, perturb the probbase and check whether imputation accuracy is reliably higher for the true probbase than for the perturbed one — if not, the minimality claim is empirically false.
Extended reading notes
Core claim
The paper's central claim is Eq. (11): the probbase q*_A that minimizes the imputation cross-entropy I_A(qhat) is approximately the true probbase q. Concretely, if one deletes all answers in a block of questions, uses the candidate probbase to compute a posterior over causes from the remaining answers, predicts the deleted answers, and measures cross-entropy against the real ones, this score is minimized when the probbase is correct. Theorem 1 proves that no other probbase and algorithm combination can beat the correct probbase under block-conditional independence, provided the VA algorithm returns the true posterior when the probbase is correct. Theorem 2 adds a strong 'if and only if' cali
Load-bearing premise
The whole method rests on Assumption 1: that the interview questions can be partitioned into blocks such that answers in different blocks are conditionally independent given the true cause of death; the paper learns those blocks from the very data it evaluates on, so misspecification is never tested.
Editorial extensions
If this is right
- Probbase verification without labels: VA programmes could check a probbase against a corpus of completed questionnaires, reducing reliance on physician-coded training data.
- Partial correction: the gradient of imputation accuracy with respect to probbase entries gives a signal for whether an entry is too high or too low, so entries can be prioritised for expert review.
- Algorithm assessment: with a trusted probbase, the same imputation score can compare VA algorithms, since well-calibrated algorithms give better imputation.
- Adaptation to new questionnaires and populations: probbases differ across populations and change with questionnaire revisions; unlabelled calibration offers a way to update them where labelled data is scarce.
- Formal identifiability: three conditionally independent blocks of size at least the number of causes are claimed to be necessary and sufficient for calibration, delineating when the approach can work.
Reading between the lines
- The approach is a form of self-supervised learning; a natural extension is to use the imputation objective as a training loss to re-estimate a full probbase from scratch in large unlabelled corpora, a large-data limit the paper mentions but does not test.
- The block-conditional-independence assumption is learned from the same simulated data used for evaluation; practical value likely hinges on whether stable question blocks transfer across populations, which is testable with multi-site datasets.
- The simulation shows high-magnitude gradient values flag perturbed entries almost perfectly, so a two-stage procedure (flag then verify only flagged entries) could substantially cut the cost of probbase re-estimation.
- If the uniqueness conjecture fails, the method still works as a veto: it can tell you a probbase is wrong, but not certify that a probbase is right.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method to calibrate or verify the 'probbase' matrix used in verbal autopsy algorithms using only unlabelled questionnaires. The idea is to quantify imputation accuracy I_A(q̂): mask all answers in a conditionally independent block, compute a posterior over causes from the remaining answers using a candidate probbase, and score the predicted answer probabilities by cross-entropy. Theorems 1 and 2 claim that the true probbase minimizes this objective (Theorem 1) and, under a stipulated 'if and only if' calibration condition, is the unique minimizer (Theorem 2). Eq. (11) asserts that minimizing I_A recovers the true probbase. Simulations on data generated under the block-conditional independence assumption with the InterVA4 algorithm show that coarsely incorrect probbases have worse imputation accuracy and that local gradient signs can detect perturbed entries with AUC ≈ 0.75–0.84. The paper is explicitly a working paper: it lists many open theoretical and practical questions and warns against real-world use.
Significance. If the central identification claim held, the method would allow VA probbases to be checked or partially corrected without gold-standard cause-of-death labels, which is valuable for the WHO 2022 questionnaire update and for hard-to-reach populations. The paper is transparent: code is public, assumptions are stated, and limitations are acknowledged explicitly. However, the uniqueness part of the identification claim is not established for the implemented algorithm; it rests on an assumed identifiability condition, and the simulations do not rule out spurious global minima. The significance is therefore real but conditional on further theoretical and numerical work.
major comments (3)
- [§2.4, Theorem 2 and Eq. (11)] The central identification claim q*_A := arg min I_A(q̂) ≈ q is supported only by Theorem 2, whose key premise is a stipulated 'if and only if' condition: V(α_-ℓ, q̂) = P(D|A_-ℓ) for all α iff q̂_{jk} = q_{jk} for all k ∉ B_ℓ. This is an identifiability assumption, not a consequence of Assumption 1 or of any standard VA algorithm. As the authors note, InterVA4 violates the weaker complete-calibration Assumption 2, so for the algorithm used in the simulations Theorem 2's condition is not known to hold. Consequently, Theorem 1 (one-way: the true probbase is a minimizer) plus Theorem 2 under an assumed condition does not establish Eq. (11) for the implemented method; incorrect probbases may also be global minima.
- [§3.2–3.3] The simulations do not test for the existence of spurious global minima. The four probbases compared in Table 1 are hand-constructed perturbations, and the gradient analysis in Eq. (12) only evaluates one-step local changes from a starting probbase already close to q. A competing probbase far from q that also minimizes I_A would not be detected by this procedure. To support Eq. (11) empirically, the authors should report minimization runs from multiple random starting points, including chaotic probbases, or otherwise demonstrate that no other local minima are encountered. As it stands, the AUC ≈ 0.75 result supports local detection of perturbations but not the global recovery claim.
- [§3.1 and §4.2(4)] The block partition {B_ℓ} is learned from the same RandomPhysician dataset used to simulate the evaluation data, and the simulation then generates data exactly under Assumption 1 with those blocks. This means the evaluation never faces block misspecification, despite the paper's own list of open questions acknowledging that correct block specification cannot be guaranteed. Since the practical usefulness of the method depends on robustness to this, and the abstract claims 'necessary and sufficient' conditions, at least one misspecification experiment (e.g., using wrong block boundaries or a block with residual dependence) is needed before the practical claim is supported.
minor comments (6)
- [Abstract] The abstract states that 'three conditional independence blocks of size at least the number of latent categories are necessary and sufficient for calibration.' This overstates the results: Theorem 2 gives a sufficient condition under a stipulated 'if and only if' calibration assumption, and Conjecture 1 remains open. Please temper the abstract to match the proven content.
- [§2.4, Eq. (9)] In the final displayed line of Eq. (9), the subscript in P(D=d_j | A_{-B_1}=a_{-B_1}) should be A_{-B_ℓ}, not A_{-B_1}, since the block is arbitrary. This typo obscures the derivation.
- [§3.3, Table 1] The text says 'When it is slightly perturbed (I_A(hat q_3))' but the preceding definition and Table 1 use hat q_4 for the slightly perturbed probbase. Please correct the reference.
- [§3.1] The simulation description writes '(A_k | D = d_k)' but the index should be d_j, consistent with the probbase notation. This typo appears in the sentence defining the simulation of non-demographic answers.
- [§1.2] Typo: 'a known partition of of [s]' should read 'a known partition of [s]'.
- [Theorem 2 proof] The proof says 'Suppose |{B_ι}| = b_ι' but the intended quantity appears to be |B_ι|. Please clarify the notation. Also, the 'almost surely' qualifier in the theorem statement is unusual; a brief explanation of the probability space would help.
Circularity Check
No significant circularity: the core identity (Eq. 9) is a proper-scoring-rule argument under Assumption 1; uniqueness is explicitly conditional on the stated 'if and only if' calibration assumption, not smuggled in.
full rationale
The paper's central derivation is not circular. Eq. (9) is a straightforward expansion under Assumption 1: P(A_k = a_k | A_-ℓ) is written as a sum over causes of P(A_k = a_k | D) P(D | A_-ℓ), which is an identity once block-conditional independence is assumed. The imputation rule F_k (Eq. 2) is then the corresponding conditional probability when the VA algorithm returns the true posterior and the probbase is correct. Theorem 1 establishes that the correct probbase minimizes the expected cross-entropy I(ĉq); this is the standard proper-scoring-rule fact that the true conditional distribution minimizes expected log loss. It is an optimization property, not a fitted parameter renamed as a prediction. Theorem 2 does rely on a strong identifiability condition—the 'if and only if' assumption that the VA posterior is correct exactly when the off-block probbase is correct—but the paper states this condition explicitly, discusses when it is violated ('This condition is violated if, for instance, the V A algorithm ignores the values q̂ entirely'), and leaves the stronger global uniqueness as Conjecture 1. That is an under-support/correctness limitation, not circularity. The simulations compare hand-constructed probbases and use finite-difference gradients on data simulated under the same block structure; this is a realistic-simulation validity concern and does not reduce the claimed prediction to the fitted data. The only self-citation (D'Ambruoso et al. 2021, with coauthor Cowan) is contextual and not load-bearing. The paper itself repeatedly warns that the method is not yet thoroughly tested and should not be used in practice, which further supports the assessment that no circular derivation is being concealed.
Assumptions & free parameters
free parameters (4)
- Block partition {B_ℓ} =
learned from RandomPhysician data in simulation; user-specified in practice
- ε (gradient step) =
1/100
- Perturbation magnitude =
0.2
- Probbase inclusion threshold =
0.2
assumptions (5)
- domain assumption Assumption 1: answers in different blocks are conditionally independent given cause of death (Eq. 8)
- domain assumption Missingness in interview answers is non-informative / MAR
- domain assumption Assumption 2 (complete calibration): a VA algorithm returns the true posterior P(D|A_{non-NA}) when the probbase is correct
- ad hoc to paper Theorem 2's 'if and only if' condition: V(α_-ℓ, \hat{q}) = P(D|A_-ℓ) iff \hat{q}=q on all k not in B_ℓ
- domain assumption Simulation data-generating model: latent Gaussian variables z_ℓ|D ~ N(0,Σ_{jℓ}) with probbase probits (Section 3.1)
Cite this review
Pith. "Pith review of Calibration of answer probabilities in verbal autopsies: Working Paper." pith.science (2026). https://pith.science/paper/GEACYSJD
@misc{pith2026251010065,
author = {Pith},
title = {Pith review of: Calibration of answer probabilities in verbal autopsies: Working Paper},
year = {2026},
howpublished = {\url{https://pith.science/paper/GEACYSJD}},
note = {Machine review of arXiv:2510.10065}
}
read the original abstract
We consider the problem of calibrating a posterior estimator of a categorical latent variable given a fixed-length ternary string using only unlabelled observations and without a specified likelihood. We primarily consider a setting in which the estimator is parametrised by estimated conditional probabilities of elements of the string given the latent variable, with calibration if the estimates are correct. Our motivating application is the `Verbal Autopsy' procedure, whereby a cause of death is probabilistically inferred following a structured interview with associates of the deceased. More generally, our setting applies to circumstances where experts can more readily describe posterior beliefs than likelihoods, due to similarity with diagnostic practices. We argue combinatorially that in general the problem is intractable without a simplifying assumption on the data distribution, though some posterior estimators can be ruled out as incompatible. We propose an assumption of block-conditional independence on substrings, allowing calibration procedures based on substring frequency, imputation, and pairwise and three-way distributions of string elements. We give theoretical results on identifiability, on consistency for distributions of either fixed support or fixed entropy, and on robustness to assumptions, finding essentially that three conditional independence blocks of size at least the number of latent categories are necessary and sufficient for calibration. We empirically evaluate methods on data simulated to resemble realistic verbal autopsy questionnaires, and find substantial promise for the approach in the practical problem of calibrating posterior estimates for causes of death.
Figures
Reference graph
Works this paper leans on
-
[1]
Byass, P. (2010). The imperfect world of global health estimates. PLoS medicine , 7(11):e1001006
2010
-
[2]
L., and Van Minh, H
Byass, P., Huong, D. L., and Van Minh, H. (2003). A probabilistic approach to interpreting Verbal Autopsies : methodology and preliminary validation in Vietnam . Scandinavian Journal of Public Health , 31(62\_suppl):32--37
2003
-
[3]
Byass, P., Hussain-Alkhateeb, L., D’Ambruoso, L., Clark, S., Davies, J., Fottrell, E., Bird, J., Kabudula, C., Tollman, S., Kahn, K., et al. (2019). An integrated approach to processing who-2016 Verbal Autopsy data: the interva-5 model. BMC medicine , 17:1--12
2019
-
[4]
Chu, Y. (2025). Leveraging language models and machine learning in Verbal Autopsy analysis. arXiv preprint arXiv:2508.19274
arXiv 2025
-
[5]
Clark, S. J., Li, Z., and McCormick, T. H. (2018). Quantifying the contributions of training data and algorithm logic to the performance of automated cause-assignment algorithms for Verbal Autopsy . arXiv preprint arXiv:1803.07141
arXiv 2018
-
[6]
D’Ambruoso, L., Price, J., Cowan, E., Goosen, G., Fottrell, E., Herbst, K., Van Der Merwe, M., Sigudla, J., Davies, J., and Kahn, K. (2021). Refining circumstances of mortality categories (comcat): a Verbal Autopsy model connecting circumstances of deaths with outcomes for public health decision-making. Global health action , 14(sup1):2000091
2021
-
[7]
Fottrell, E., Kahn, K., Tollman, S., and Byass, P. (2011). Probabilistic methods for Verbal Autopsy interpretation: Interva robustness in relation to variations in a priori probabilities. PLoS One , 6(11):e27200
2011
-
[8]
J., Morof, D., Joubert, J
Groenewald, P., Thomas, J., Clark, S. J., Morof, D., Joubert, J. D., Kabudula, C., Li, Z., and Bradshaw, D. (2023). Agreement between cause of death assignment by computer-coded Verbal Autopsy methods and physician coding of Verbal Autopsy interviews in South Africa . Global Health Action , 16(1):2285105
2023
Show all 22 references
-
[9]
James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An introduction to statistical learning: with applications in R , volume 103. Springer
2013
-
[10]
R., McComick, T
Li, Z. R., McComick, T. H., and Clark, S. J. (2019). Using Bayesian latent Gaussian graphical models to infer symptom associations in Verbal Autopsies . Bayesian analysis , 15(3):781
2019
-
[11]
R., McCormick, T
Li, Z. R., McCormick, T. H., and Clark, S. J. (2014). InterVA4 : An R package to analyze Verbal Autopsy data. Center for Statistics and the Social Sciences, University of Washington: Vienna, Austria: R Foundation for Statistical Computing.[1044]
2014
-
[12]
R., Thomas, J., Choi, E., McCormick, T
Li, Z. R., Thomas, J., Choi, E., McCormick, T. H., and Clark, S. J. (2023). The openva toolkit for Verbal Autopsies . the R Journal
2023
-
[13]
R., Thomas, J., McCormick, T
Li, Z. R., Thomas, J., McCormick, T. H., and Clark, S. J. (2025). openVA: Automated Method for Verbal Autopsy . R package version 1.2.0
2025
-
[14]
R., Wu, Z., Chen, I., and Clark, S
Li, Z. R., Wu, Z., Chen, I., and Clark, S. J. (2024). Bayesian nested latent class models for cause-of-death assignment using Verbal Autopsies across multiple domains. The annals of applied statistics , 18(2):1137
2024
-
[15]
H., Li, Z
McCormick, T. H., Li, Z. R., Calvert, C., Crampin, A. C., Kahn, K., and Clark, S. J. (2016). Probabilistic cause-of-death assignment using Verbal Autopsies . Journal of the American Statistical Association , 111(515):1036--1049
2016
-
[16]
Y., Alam, D., Tollman, S., Samarikhalaj, A., and Jha, P
Miasnikof, P., Giannakeas, V., Gomes, M., Aleksandrowicz, L., Shestopaloff, A. Y., Alam, D., Tollman, S., Samarikhalaj, A., and Jha, P. (2015). Naive Bayes classifiers for Verbal Autopsies : comparison to physician-based classification for 21,000 child and adult deaths. BMC me...
2015
-
[17]
J., Lopez, A
Murray, C. J., Lopez, A. D., Black, R., Ahuja, R., Ali, S. M., Baqui, A., Dandona, L., Dantzer, E., Das, V., Dhingra, U., et al. (2011). Population health metrics research consortium gold standard Verbal Autopsy validation study: design, implementation, and development of anal...
2011
-
[18]
L., Flaxman, A
Serina, P., Riley, I., Stewart, A., James, S. L., Flaxman, A. D., Lozano, R., Hernandez, B., Mooney, M. D., Luning, R., Black, R., et al. (2015). Improving performance of the tariff method for assigning causes of death to Verbal Autopsies . BMC medicine , 13:1--13
2015
-
[19]
World Health Organization et al. (2012). Verbal Autopsy standards: the 2012 who Verbal Autopsy instrument. Geneva: WHO
2012
-
[20]
World Health Organization et al. (2022). Verbal Autopsy standards: the 2022 who Verbal Autopsy instrument. Geneva: WHO
2022
-
[21]
and Li, Z
Zhu, Y. and Li, Z. R. (2024). Hierarchical latent class models for mortality surveillance using partially verified Verbal Autopsies . arXiv preprint arXiv:2410.09274
2024 arXiv
-
[22]
and Li, Z
Zhu, Y. and Li, Z. R. (2025). Flexible Bayesian tensor decomposition for Verbal Autopsy data. arXiv preprint arXiv:2502.00171
2025 arXiv
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.