REVIEW 5 major objections 5 minor 27 references
LAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read An off-the-shelf LLM, prompted with age-specific cause lists, can assign verbal-autopsy causes of death 5–10% more accurately than standard automated algorithms.
desk verdict A useful proof-of-concept with real code and data, but the headline '5-10% over baselines' is an adult-only result presented as cross-age; child and neonate numbers lack any comparator. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a structured prompting protocol: for each age group, the model receives an explicit list of allowed causes, instructions to build an internal onset-to-death timeline, a primary-cause hierarchy (immediate mechanism, underlying disease, contributing factors), and few-shot examples; it must output one cause plus a rationale. This restricts a general LLM to the study's label space and encodes clinical heuristics without retraining. The second load-bearing piece is post-hoc calibration: the model's top-five probabilities are reweighted with nonincreasing weights $\alpha_1 \ge \cdots \ge \alpha_5$ fitted by minimizing the gap between predicted and training cause distributions, so population-level estimates improve while individual rankings stay unchanged.
What would settle it
Run GPT-5 on the same verbal autopsy records with the free-text narrative removed, keeping only structured symptom answers and demographics; if its accuracy falls to LCVA's level or below, the reported 5–10% gain comes from reading narratives, not from reasoning over symptoms.
Extended reading notes
Core claim
The paper claims that GPT-5, used as a frozen commercial model with hand-written age-specific prompts, assigns the correct cause of death more often than the LCVA baseline on the PHMRC verbal autopsy reference set. In leave-one-site-out testing, top-1 accuracies were 48.6% for adults with 34 causes, 50.5% for children with 21 causes, and 53.5% for neonates with 6 causes, a 5–10 percentage-point gain over the traditional baseline. A lightweight calibration step reweights the model's top-five predicted probabilities against training prevalence and raises adult cause-specific mortality fraction accuracy from 0.73 to 0.77 without changing top-N accuracy. Accuracy grows with narrative length, from 49.4% for short narratives to 63.2% for long ones, and the largest gains are for causes with distinctive presentations such as maternal death and road-traffic injury.
Load-bearing premise
The result assumes a fair comparison: the LLM sees the free-text narrative while the traditional baseline sees only structured symptoms, so the 5–10% gap may be an input-modality effect rather than better diagnostic reasoning.
Editorial extensions
If this is right
- Deploying an LLM with the paper's prompting template could lift individual cause-of-death accuracy in verbal-autopsy surveillance by 5–10 percentage points over current automated coding, with no training data from the target site.
- The calibration step improves cause-specific mortality fractions, raising the adult mean from 0.73 to 0.77 without changing top-N rankings, making it a cheap way to improve population-level estimates.
- The method carries over to children and neonates with only a prompt change, whereas the LCVA baseline is not available for neonates, lowering the engineering cost of expanding surveillance.
- Gains concentrate on causes with distinctive presentations—maternal deaths above 90% and road-traffic injuries around 73%—so estimates for those causes can be most trusted.
- Longer narratives predict higher accuracy, suggesting that investments in free-text collection quality directly improve LLM-based verbal autopsy performance.
Reading between the lines
- A like-for-like rerun—GPT-5 on structured symptom checklists only, or an embedding classifier on narrative plus symptoms—could shrink or eliminate the reported 5–10% advantage, because the current comparison confounds model skill with access to narrative text.
- If narrative access is the true driver, then verbal autopsy instruments should be redesigned to encourage open-ended descriptions, and structured-only historical datasets will understate what LLMs can do.
- The same top-five reweighting calibration applies to any ranked classifier, so its population-level benefit is likely portable beyond GPT-5.
- PHMRC narratives are largely short English transcripts from a historical validation study, so multilingual and noisier real-world transcripts may not reproduce the same gains until separately validated.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LA-VA, a pipeline that combines GPT-5 predictions, an LCVA Bayesian baseline, embedding-based classifiers, and stacked ensembles for verbal-autopsy cause-of-death assignment, evaluated on the PHMRC dataset across adult (N=7,580), child (N=1,960), and neonate (N=2,438) modules under leave-one-site-out cross-validation. The authors report that GPT-5 achieves the highest individual top-1 accuracy (48.6% adult, 50.5% child, 53.5% neonate) and, after a calibration step that reweights top-5 predictions toward training prevalence, improves CSMF accuracy relative to LCVA. The central claim is that off-the-shelf LLM use can outperform traditional automated VA algorithms by 5–10%.
Significance. If substantiated, the result would be practically useful for mortality surveillance in settings without medical certification. Strengths of the paper include its use of a well-established reference dataset, a clearly specified leave-one-site-out evaluation that targets distribution shift, an explicit and reproducible calibration procedure, and public code. The main limitations are that the 5–10% improvement over traditional baselines is established only for adults, that the child and neonate modules have no traditional baseline at all, and that GPT-5's advantage is confounded with its access to free-text narratives that the LCVA baseline does not receive. The narrative-denominator inconsistency and the absence of uncertainty quantification further weaken the quantitative claims.
major comments (5)
- [Abstract and §2 vs. Appendix B] The abstract and Discussion claim that GPT-5 outperforms traditional statistical machine learning baselines by 5–10% across all three age groups, but the only quantitative comparison against a traditional baseline (LCVA) is for adults in §2 and Table 1. Appendix B states that for children 'independent baselines are scarce' and that LCVA and related probabilistic models do not support neonates. The 5–10% margin is therefore unmeasured for child and neonate modules; the headline claim should be restricted to adults or supported by baselines for the other age groups.
- [§2, 'Narrative Length and Accuracy'] The text states that narratives were present in 87.1% of cases (9,113/10,466), but §1.1 gives the analyzed cohort as 11,978 cases (7,580 + 1,960 + 2,438). Since 9,113/10,466 ≈ 87.1% while 9,113/11,978 ≈ 76.1%, the reported total and the narrative denominator are inconsistent. This needs reconciliation because the subsequent narrative-length accuracy analysis depends on which denominator is correct.
- [§2, Figure 1, Table 1] No confidence intervals, standard errors, or significance tests are reported for the top-1 accuracy comparisons between GPT-5 and LCVA, and the leave-one-site-out design has only six test folds. The point estimates alone do not establish that GPT-5's advantage over LCVA, which is the basis of the abstract's central claim, is statistically reliable.
- [§1.2, pipeline components] The comparison between GPT-5 and LCVA is confounded by input modality: GPT-5 receives demographics, structured symptoms, and the free-text narrative, while LCVA uses only structured symptom data. The reported 5–10% advantage could therefore reflect access to the narrative rather than the language model's reasoning. The paper should either run GPT-5 on structured symptoms alone or explicitly frame the comparison as 'narrative-augmented LLM vs. structured-only baseline' rather than as evidence of LLM superiority.
- [§1.4, Eq. (1), Table 1] The calibrated-GPT-5 CSMF comparison with LCVA is not apples-to-apples: the alpha weights are fitted to make the calibrated predictions match the training distribution, so the resulting CSMF improvement is partly a post-stratification adjustment by construction. This does not affect the top-1 accuracy claim, but the paper should state clearly that the CSMF comparison reflects calibration toward training prevalence, not an intrinsic population-level advantage of the uncalibrated model.
minor comments (5)
- [Appendix A.2] The text says 'full guidance appears in Appendix Z', but no Appendix Z is included in the manuscript; the reference should be updated or the material provided.
- [Appendix A.1] The system prompt instructs the model to 'Think step-by-step (hide the chain-of-thought)', which is internally inconsistent; the intended instruction should be clarified.
- [§2, 'Narrative Length and Accuracy'] The reported p<0.001 for the trend in accuracy with narrative length is not accompanied by a description of the statistical test used, so the reader cannot assess whether the comparison accounts for clustering by site or cause.
- [Appendix B, Figures 2 and 3] The figure legends label the plots 'No GPT-4o', which is unexplained and does not match the methods described in the main text; the legends should be corrected or removed.
- [Table 1] The 'Adult Mean' row reports values such as 0.73 (0.1) without defining the parenthetical quantity; if it is the standard deviation across sites, this should be stated, and confidence intervals should be given for the mean differences discussed in the text.
Circularity Check
No significant circularity: GPT-5 performance is an empirical benchmark; the CSMF calibration is fitted but transparently labeled and does not affect the top-1 headline.
full rationale
The paper's central claim is an empirical benchmark of GPT-5 against LCVA and embedding classifiers on the PHMRC dataset, not a derivation from first principles. The only fitted component is the Section 1.4 calibration, where alpha weights are estimated on the training set to match the training cause distribution. This is a standard post-hoc calibration; the paper explicitly states that it has no impact on Top-N accuracy, and the headline top-1 accuracies are independent of it. The calibrated CSMF gains in Table 1 are not model 'predictions' but training-set-calibrated estimates evaluated under leave-one-site-out cross-validation, so no equation reduces to its own input by construction. Self-citations (e.g., Datta et al. 2021; Fiksel et al. 2022, 2023; Pramanik et al. 2025) support background claims and the calibration idea but are not load-bearing; they are peer-reviewed and externally falsifiable. Appendix B explicitly flags that child baselines are scarce and LCVA does not support neonates, meaning the abstract's '5-10% over traditional baselines' is, on the reported evidence, an Adult-only comparison. This is a generalization/correctness weakness, not a circularity, so it does not raise the circularity score. No circular step is identified.
Assumptions & free parameters
free parameters (3)
- Calibration weights alpha_1..alpha_5 (and per confidence level) =
Not reported; estimated via convex optimization on training set
- Weighted ensemble weights =
Grid-searched
- Hyperparameters for logistic regression, random forest, XGBoost =
Tuned via nested cross-validation
assumptions (4)
- domain assumption PHMRC physician-certified causes are the gold standard.
- domain assumption GPT-5 is a stable, accessible model with reproducible outputs.
- domain assumption Narratives in PHMRC are in English and sufficient for classification.
- domain assumption LCVA is a fair and correctly implemented baseline.
Cite this review
Pith. "Pith review of LAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination." pith.science (2026). https://pith.science/paper/PIC64PPY
@misc{pith2026250909602,
author = {Pith},
title = {Pith review of: LAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination},
year = {2026},
howpublished = {\url{https://pith.science/paper/PIC64PPY}},
note = {Machine review of arXiv:2509.09602}
}
read the original abstract
Verbal autopsy (VA) is a critical tool for estimating causes of death in resource-limited settings where medical certification is unavailable. This study presents LA-VA, a proof-of-concept pipeline that combines Large Language Models (LLMs) with traditional algorithmic approaches and embedding-based classification for improved cause-of-death prediction. Using the Population Health Metrics Research Consortium (PHMRC) dataset across three age categories (Adult: 7,580; Child: 1,960; Neonate: 2,438), we evaluate multiple approaches: GPT-5 predictions, LCVA baseline, text embeddings, and meta-learner ensembles. Our results demonstrate that GPT-5 achieves the highest individual performance with average test site accuracies of 48.6% (Adult), 50.5% (Child), and 53.5% (Neonate), outperforming traditional statistical machine learning baselines by 5-10%. Our findings suggest that simple off-the-shelf LLM-assisted approaches could substantially improve verbal autopsy accuracy, with important implications for global health surveillance in low-resource settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Extracting cause of death from verbal autopsy with deep learning interpretable methods
Alberto Blanco, Alicia P \'e rez, Arantza Casillas, and Daniel Cobos. Extracting cause of death from verbal autopsy with deep learning interpretable methods. IEEE Journal of Biomedical and Health Informatics, 25 0 (4): 0 1315--1325, 2020
work page 2020
-
[2]
Strengthening standardised interpretation of verbal autopsy data: the new interva-4 tool
Peter Byass, Daniel Chandramohan, Samuel J Clark, Lucia D'ambruoso, Edward Fottrell, Wendy J Graham, Abraham J Herbst, Abraham Hodgson, Sennen Hounton, Kathleen Kahn, et al. Strengthening standardised interpretation of verbal autopsy data: the new interva-4 tool. Global Health Action, 5 0 (1): 0 19281, 2012
work page 2012
-
[3]
Leveraging Language Models and Machine Learning in Verbal Autopsy Analysis
Yue Chu. Leveraging language models and machine learning in verbal autopsy analysis. arXiv preprint arXiv:2508.19274, 2025
work page Pith review arXiv 2025
-
[4]
Regularized bayesian transfer learning for population-level etiological distributions
Abhirup Datta, Jacob Fiksel, Agbessi Amouzou, and Scott L Zeger. Regularized bayesian transfer learning for population-level etiological distributions. Biostatistics, 22 0 (4): 0 836--857, 2021
work page 2021
-
[5]
Cvxpy: A python-embedded modeling language for convex optimization
Steven Diamond and Stephen Boyd. Cvxpy: A python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17 0 (83): 0 1--5, 2016
2016
-
[6]
Tamer H. Farag, Jeffrey P. Koplan, Robert F. Breiman, Shabir A. Madhi, Penny M. Heaton, Trevor Mundel, Jaume Ordi, Quique Bassat, Clara Menendez, and Scott F. Dowell. Precisely tracking childhood death. American Journal of Tropical Medicine and Hygiene, 97 0 (1): 0 3--6, 2017. doi:10.4269/ajtmh.16-0952
-
[7]
Generalized bayes quantification learning under dataset shift
Jacob Fiksel, Abhirup Datta, Agbessi Amouzou, and Scott Zeger. Generalized bayes quantification learning under dataset shift. Journal of the American Statistical Association, 117 0 (540): 0 2163--2181, 2022
work page 2022
-
[8]
Correcting for verbal autopsy misclassification bias in cause-specific mortality estimates
Jacob Fiksel, Brian Gilbert, Emily Wilson, Henry Kalter, Almamy Kante, Aveika Akum, Dianna Blau, Quique Bassat, Ivalda Macicame, Eduardo Samo Gudo, et al. Correcting for verbal autopsy misclassification bias in cause-specific mortality estimates. The American Journal of Tropical Medicine and Hygiene, 108 0 (5\_Suppl): 0 66--74, 2023. doi:10.4269/ajtmh.22-...
Show all 27 references
-
[9]
Multi-cause calibration of verbal autopsy--based cause-specific mortality estimates of children and neonates in mozambique
Brian Gilbert, Jacob Fiksel, Emily Wilson, Henry Kalter, Almamy Kante, Aveika Akum, Dianna Blau, Quique Bassat, Ivalda Macicame, Eduardo Samo Gudo, et al. Multi-cause calibration of verbal autopsy--based cause-specific mortality estimates of children and neonates in mozambique...
2023
-
[10]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[11]
Performance of the tariff method for assigning causes of death from verbal autopsy: validation study using clinical diagnostic gold standards
Spencer L James, Abraham D Flaxman, and Christopher JL Murray. Performance of the tariff method for assigning causes of death from verbal autopsy: validation study using clinical diagnostic gold standards. PLoS Medicine, 8 0 (1): 0 e1000326, 2011
2011
-
[12]
Dspy: Compiling declarative language model calls into state-of-the-art pipelines
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth International Co...
2024
-
[13]
Bayesian analysis of verbal autopsy data using factor models with age-and sex-dependent associations between symptoms
Tsuyoshi Kunihama, Zehang Richard Li, Samuel J Clark, and Tyler H McCormick. Bayesian analysis of verbal autopsy data using factor models with age-and sex-dependent associations between symptoms. The Annals of Applied Statistics, 19 0 (3): 0 2237--2253, 2025
2025
-
[14]
Bayesian nested latent class models for cause-of-death assignment using verbal autopsies across multiple domains
Zehang Richard Li, Zhenke Wu, Irena Chen, and Samuel J Clark. Bayesian nested latent class models for cause-of-death assignment using verbal autopsies across multiple domains. The Annals of Applied Statistics, 18 0 (2): 0 1137, 2024
2024
-
[15]
Countrywide mortality surveillance for action in mozambique: results from a national sample-based vital statistics system for mortality and cause of death
Ivalda Macicame, Almamy M Kante, Emily Wilson, Brian Gilbert, Alain Koffi, Sheila Nhachungue, Celso Monjane, Pedro Duce, Antonio Adriano, Sergio Chicumbe, et al. Countrywide mortality surveillance for action in mozambique: results from a national sample-based vital statistics ...
2023
-
[16]
Improving cause-of-death classification from verbal autopsy reports
Thokozile Manaka, Terence van Zyl, and Deepak Kar. Improving cause-of-death classification from verbal autopsy reports. In Southern African Conference for Artificial Intelligence Research, pages 46--59. Springer, 2022
2022
-
[17]
Multi-step transfer learning in natural language processing for the health domain
Thokozile Manaka, Terence Van Zyl, Deepak Kar, and Alisha Wade. Multi-step transfer learning in natural language processing for the health domain. Neural Processing Letters, 56 0 (3): 0 177, 2024
2024
-
[18]
Probabilistic cause-of-death assignment using verbal autopsies
Tyler H McCormick, Zehang Richard Li, Clara Calvert, Amelia C Crampin, Kathleen Kahn, and Samuel J Clark. Probabilistic cause-of-death assignment using verbal autopsies. Journal of the American Statistical Association, 111 0 (515): 0 1036--1049, 2016
2016
-
[19]
Population health metrics research consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasets
Christopher JL Murray, Alan D Lopez, Robert Black, Ravindra Ahuja, Saira Aleem, Abdullah Baqui, Nazma Begum, Zulfiqar Bhutta, Lalit Dandona, Eric Dantzer, et al. Population health metrics research consortium gold standard verbal autopsy validation study: design, implementation...
2011
-
[20]
Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies
Christopher JL Murray, Rafael Lozano, Abraham D Flaxman, Alireza Vahdatpour, and Alan D Lopez. Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. Population Health Metrics, 9 0 (1): 0 28, 2011 b
2011
-
[21]
Using verbal autopsy to measure causes of death: the comparative performance of existing methods
Christopher JL Murray, Rafael Lozano, Abraham D Flaxman, Peter Serina, David Phillips, Andrea Stewart, Spencer L James, Alireza Vahdatpour, Charles Atkinson, Michael K Freeman, et al. Using verbal autopsy to measure causes of death: the comparative performance of existing meth...
2014
-
[22]
Modeling structure and country-specific heterogeneity in misclassification matrices of verbal autopsy-based cause of death classifiers
Sandipan Pramanik, Scott Zeger, Dianna Blau, and Abhirup Datta. Modeling structure and country-specific heterogeneity in misclassification matrices of verbal autopsy-based cause of death classifiers. The Annals of Applied Statistics, 19 0 (2): 0 1214--1239, 2025
2025
-
[23]
Toward expert-level medical question answering with large language models
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, 31 0 (3): 0 943--950, 2025
2025
-
[24]
Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint a...
2023 arXiv
-
[25]
Text embeddings --- voyage ai documentation
Voyage AI . Text embeddings --- voyage ai documentation. https://docs.voyageai.com/docs/embeddings, 2025. Accessed: 2025-09-07
2025
-
[26]
Bayesian active questionnaire design for cause-of-death assignment using verbal autopsies
Toshiya Yoshida, Trinity Shuxian Fan, Tyler McCormick, Wu Zhenke, and Zehang Richard Li. Bayesian active questionnaire design for cause-of-death assignment using verbal autopsies. In Conference on Health, Inference, and Learning, pages 37--49. PMLR, 2023
2023
-
[27]
Fine-tuning language models from human preferences
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019
1909 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.