{"id":"d8ae7616-04e6-482a-b60a-6418b5ca1cab","arxiv_id":"2509.09602","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GPT-5 outperforms traditional verbal autopsy algorithms on cause-of-death prediction, achieving 48.6% to 53.5% top-1 accuracy across adult, child, and neonate datasets.","lead":"This paper tests whether a large language model can determine cause of death from verbal autopsy interviews more accurately than existing algorithms. On a six-site global dataset, GPT-5 tops out at 48-54% correct, beating traditional baselines by 5-10%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"5-10% advantage over traditional baselines is documented only for Adults; Appendix B states Child baselines are scarce and LCVA does not support Neonates, so the abstract over-generalizes.","rationale":"The reader's verdict is CONDITIONAL, and this stress-test supports that verdict without moving it. The most load-bearing problem is not the modality confound alone, though that is real and was the reader's named weakest assumption. The sharper gap is that the paper's headline comparison to traditional baselines has no measured comparator for Child or Neonate. Appendix B is explicit about this, and it is a limitation asserted within the manuscript itself, so it must be weighed directly. Because the abstract reports 'average test site accuracies' for all three age groups immediately before claiming a 5-10% improvement over traditional baselines, readers will naturally take the improvement as applying to all three. The evidence only supports it for Adults. This is a scope-of-claim error rather than a fatal flaw: the Adult comparison, code availability, and public dataset are real assets, and the direction of the Adult result is plausible. The right fix is to either run suitable child/neonate baselines or rephrase the abstract and discussion to state that the outperformance claim is Adult-only. The modality confound remains a secondary concern that would also benefit from an ablation with structured symptoms only, but it does not need to be resolved before the baseline gap is addressed. A concrete check running Tariff or InterVA on the child and neonate modules would settle whether the cross-age generalization is supportable.","tokens_in":11420,"tokens_out":4262,"duration_ms":37444,"concrete_test":"Run an established VA algorithm with child and neonate support (e.g., Tariff 2.0, InterVA-4, or an adapted LCVA) on the same PHMRC child/neonate subsets with the same leave-one-site-out folds and cause lists, and compare mean test-site Top-1 accuracy and site-level spread against GPT-5. If the GPT-5 margin over this baseline is within site-level noise or reverses for Child/Neonate, the abstract's '5-10%' claim should be restricted to Adults.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim presents GPT-5 as 'outperforming traditional statistical machine learning baselines by 5-10%' as if this held across Adult, Child, and Neonate. In the reported experiments, the only quantitative comparison against a traditional baseline is for Adults: Section 2 and Table 1 compare GPT-5 with LCVA on the adult module. Appendix B explicitly says that for children 'most traditional probabilistic models require pediatric-specific modifications and have not been benchmarked separately,' and for neonates 'LCVA and related probabilistic models do not currently support neonates.' Therefore the 5-10% margin is unmeasured for two of the three age groups; the headline result is, on the evidence given, an Adult-only claim. The reader's modality confound is a related but distinct gap: even within Adults, LCVA is trained on structured symptom responses while GPT-5 receives demographics, symptoms, and free-text narrative, so the marginal contribution of the LLM's reasoning is not separated from its access to the narrative. The missing child/neonate baselines is the more direct weakness because the abstract's cross-age generalization lacks any comparator for those modules.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LA-VA, a pipeline that combines GPT-5 predictions, an LCVA Bayesian baseline, embedding-based classifiers, and stacked ensembles for verbal-autopsy cause-of-death assignment, evaluated on the PHMRC dataset across adult (N=7,580), child (N=1,960), and neonate (N=2,438) modules under leave-one-site-out cross-validation. The authors report that GPT-5 achieves the highest individual top-1 accuracy (48.6% adult, 50.5% child, 53.5% neonate) and, after a calibration step that reweights top-5 predictions toward training prevalence, improves CSMF accuracy relative to LCVA. The central claim is that off-the-shelf LLM use can outperform traditional automated VA algorithms by 5–10%.","tokens_in":11794,"tokens_out":3997,"duration_ms":33758,"significance":"If substantiated, the result would be practically useful for mortality surveillance in settings without medical certification. Strengths of the paper include its use of a well-established reference dataset, a clearly specified leave-one-site-out evaluation that targets distribution shift, an explicit and reproducible calibration procedure, and public code. The main limitations are that the 5–10% improvement over traditional baselines is established only for adults, that the child and neonate modules have no traditional baseline at all, and that GPT-5's advantage is confounded with its access to free-text narratives that the LCVA baseline does not receive. The narrative-denominator inconsistency and the absence of uncertainty quantification further weaken the quantitative claims.","major_comments":[{"comment":"The abstract and Discussion claim that GPT-5 outperforms traditional statistical machine learning baselines by 5–10% across all three age groups, but the only quantitative comparison against a traditional baseline (LCVA) is for adults in §2 and Table 1. Appendix B states that for children 'independent baselines are scarce' and that LCVA and related probabilistic models do not support neonates. The 5–10% margin is therefore unmeasured for child and neonate modules; the headline claim should be restricted to adults or supported by baselines for the other age groups.","section":"Abstract and §2 vs. Appendix B"},{"comment":"The text states that narratives were present in 87.1% of cases (9,113/10,466), but §1.1 gives the analyzed cohort as 11,978 cases (7,580 + 1,960 + 2,438). Since 9,113/10,466 ≈ 87.1% while 9,113/11,978 ≈ 76.1%, the reported total and the narrative denominator are inconsistent. This needs reconciliation because the subsequent narrative-length accuracy analysis depends on which denominator is correct.","section":"§2, 'Narrative Length and Accuracy'"},{"comment":"No confidence intervals, standard errors, or significance tests are reported for the top-1 accuracy comparisons between GPT-5 and LCVA, and the leave-one-site-out design has only six test folds. The point estimates alone do not establish that GPT-5's advantage over LCVA, which is the basis of the abstract's central claim, is statistically reliable.","section":"§2, Figure 1, Table 1"},{"comment":"The comparison between GPT-5 and LCVA is confounded by input modality: GPT-5 receives demographics, structured symptoms, and the free-text narrative, while LCVA uses only structured symptom data. The reported 5–10% advantage could therefore reflect access to the narrative rather than the language model's reasoning. The paper should either run GPT-5 on structured symptoms alone or explicitly frame the comparison as 'narrative-augmented LLM vs. structured-only baseline' rather than as evidence of LLM superiority.","section":"§1.2, pipeline components"},{"comment":"The calibrated-GPT-5 CSMF comparison with LCVA is not apples-to-apples: the alpha weights are fitted to make the calibrated predictions match the training distribution, so the resulting CSMF improvement is partly a post-stratification adjustment by construction. This does not affect the top-1 accuracy claim, but the paper should state clearly that the CSMF comparison reflects calibration toward training prevalence, not an intrinsic population-level advantage of the uncalibrated model.","section":"§1.4, Eq. (1), Table 1"}],"minor_comments":[{"comment":"The text says 'full guidance appears in Appendix Z', but no Appendix Z is included in the manuscript; the reference should be updated or the material provided.","section":"Appendix A.2"},{"comment":"The system prompt instructs the model to 'Think step-by-step (hide the chain-of-thought)', which is internally inconsistent; the intended instruction should be clarified.","section":"Appendix A.1"},{"comment":"The reported p<0.001 for the trend in accuracy with narrative length is not accompanied by a description of the statistical test used, so the reader cannot assess whether the comparison accounts for clustering by site or cause.","section":"§2, 'Narrative Length and Accuracy'"},{"comment":"The figure legends label the plots 'No GPT-4o', which is unexplained and does not match the methods described in the main text; the legends should be corrected or removed.","section":"Appendix B, Figures 2 and 3"},{"comment":"The 'Adult Mean' row reports values such as 0.73 (0.1) without defining the parenthetical quantity; if it is the standard deviation across sites, this should be stated, and confidence intervals should be given for the mean differences discussed in the text.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible proof-of-concept, but the abstract overstates the breadth of the evidence. The most important revisions are to restrict the cross-age claim to the comparisons actually performed, correct the narrative denominator inconsistency, and add uncertainty quantification or soften the top-1 comparison. The modality confound should be addressed head-on either with an ablation or with a carefully hedged interpretation. I see no evidence of fabrication; the issues are overgeneralization and incomplete reporting rather than a fundamentally unsound core idea."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate proof-of-concept that GPT-5 improves adult verbal autopsy accuracy over LCVA on the PHMRC benchmark, but the abstract's '5-10% over traditional baselines' is an adult-only claim dressed up as cross-age. The child and neonate results have no quantitative baseline at all, as the authors admit in Appendix B.\n\nWhat's genuinely useful: they evaluate GPT-5 with leave-one-site-out CV on a standard reference dataset, compare against LCVA, embedding classifiers, and ensembles, and they release code. The calibration step is simple and clearly described, and it doesn't touch top-1 accuracy, so the individual-level result stands on its own. The cause-specific tables (maternal deaths, road traffic, etc.) are informative for deployment.\n\nSoft spots, in order of weight. First, the missing child/neonate baselines. The abstract says GPT-5 'outperforming traditional statistical machine learning baselines by 5-10%' as if that were measured across all three age groups. It wasn't. For children, Appendix B says traditional models 'have not been benchmarked separately'; for neonates, LCVA doesn't run. The 5-10% figure comes from the adult comparison, period. That needs to be stated in the abstract or the claim pulled back to adults.\n\nSecond, the modality confound: GPT-5 receives demographics, symptoms, and free-text narrative; LCVA receives only structured symptoms. Part of the gap is likely input information, not reasoning. The embedding classifier also reads narrative and still lags, which helps, but it's not a clean isolation of the LLM's contribution. An ablation with GPT-5 on symptoms only would sharpen the claim.\n\nThird, no confidence intervals or significance tests are reported for the headline top-1 numbers. That's a fixable gap but matters for a 5-10% claim.\n\nFourth, a small internal inconsistency: Section 1.1 says 11,978 cases; Section 2 says narratives were present in 9,113/10,466. That's an unexplained 1,512-case discrepancy. Minor but should be corrected.\n\nThe calibration step is fine: fitting alpha weights to the training distribution does make CSMF improvement partly self-fulfilling, but the paper says so, and it does not affect top-1 accuracy.\n\nBottom line: this is a solid empirical contribution for the VA/global-health ML audience, worth a serious referee, but it needs an honest abstract, an adult-only claim, and ideally an ablation and error bars.","headline":"A useful proof-of-concept with real code and data, but the headline '5-10% over baselines' is an adult-only result presented as cross-age; child and neonate numbers lack any comparator.","tokens_in":12181,"tokens_out":2230,"would_cite":true,"duration_ms":20049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An off-the-shelf LLM, prompted with age-specific cause lists, can assign verbal-autopsy causes of death 5–10% more accurately than standard automated algorithms.","keywords":["verbal autopsy","cause of death","large language models","GPT-5","PHMRC dataset","leave-one-site-out validation","CSMF accuracy","global health surveillance"],"falsifier":"Run GPT-5 on the same verbal autopsy records with the free-text narrative removed, keeping only structured symptom answers and demographics; if its accuracy falls to LCVA's level or below, the reported 5–10% gain comes from reading narratives, not from reasoning over symptoms.","tokens_in":11244,"feed_emoji":"🩺","tokens_out":11086,"duration_ms":94703,"temperature":0.7,"pith_summary":"This paper argues that a general-purpose large language model, given a verbal autopsy transcript and age-specific instructions, can outperform dedicated automated cause-of-death algorithms without any training. On a six-site reference dataset, GPT-5's top-1 accuracies were 48.6% for adults, 50.5% for children, and 53.5% for neonates, beating the LCVA probabilistic baseline by 5–10 percentage points. If true, it would let health surveillance systems in places without medical certification use narratives that are already being collected, rather than reducing interviews to symptom checklists, to get more accurate mortality estimates.","feed_headline":"GPT-5 beats standard verbal autopsy algorithms by 5-10%","feed_subtitle":"Prompted with age-specific cause lists, the model uses narratives to lift cause-of-death accuracy by 5–10 points.","key_machinery":"The carrying mechanism is a structured prompting protocol: for each age group, the model receives an explicit list of allowed causes, instructions to build an internal onset-to-death timeline, a primary-cause hierarchy (immediate mechanism, underlying disease, contributing factors), and few-shot examples; it must output one cause plus a rationale. This restricts a general LLM to the study's label space and encodes clinical heuristics without retraining. The second load-bearing piece is post-hoc calibration: the model's top-five probabilities are reweighted with nonincreasing weights $\\alpha_1 \\ge \\cdots \\ge \\alpha_5$ fitted by minimizing the gap between predicted and training cause distributions, so population-level estimates improve while individual rankings stay unchanged.","core_discovery":"The paper claims that GPT-5, used as a frozen commercial model with hand-written age-specific prompts, assigns the correct cause of death more often than the LCVA baseline on the PHMRC verbal autopsy reference set. In leave-one-site-out testing, top-1 accuracies were 48.6% for adults with 34 causes, 50.5% for children with 21 causes, and 53.5% for neonates with 6 causes, a 5–10 percentage-point gain over the traditional baseline. A lightweight calibration step reweights the model's top-five predicted probabilities against training prevalence and raises adult cause-specific mortality fraction accuracy from 0.73 to 0.77 without changing top-N accuracy. Accuracy grows with narrative length, from 49.4% for short narratives to 63.2% for long ones, and the largest gains are for causes with distinctive presentations such as maternal death and road-traffic injury.","pith_inferences":["A like-for-like rerun—GPT-5 on structured symptom checklists only, or an embedding classifier on narrative plus symptoms—could shrink or eliminate the reported 5–10% advantage, because the current comparison confounds model skill with access to narrative text.","If narrative access is the true driver, then verbal autopsy instruments should be redesigned to encourage open-ended descriptions, and structured-only historical datasets will understate what LLMs can do.","The same top-five reweighting calibration applies to any ranked classifier, so its population-level benefit is likely portable beyond GPT-5.","PHMRC narratives are largely short English transcripts from a historical validation study, so multilingual and noisier real-world transcripts may not reproduce the same gains until separately validated."],"forward_implications":["Deploying an LLM with the paper's prompting template could lift individual cause-of-death accuracy in verbal-autopsy surveillance by 5–10 percentage points over current automated coding, with no training data from the target site.","The calibration step improves cause-specific mortality fractions, raising the adult mean from 0.73 to 0.77 without changing top-N rankings, making it a cheap way to improve population-level estimates.","The method carries over to children and neonates with only a prompt change, whereas the LCVA baseline is not available for neonates, lowering the engineering cost of expanding surveillance.","Gains concentrate on causes with distinctive presentations—maternal deaths above 90% and road-traffic injuries around 73%—so estimates for those causes can be most trusted.","Longer narratives predict higher accuracy, suggesting that investments in free-text collection quality directly improve LLM-based verbal autopsy performance."],"supporting_citations":[{"why":"Supplies the gold-standard verbal autopsy reference dataset and physician-certified cause labels on which every accuracy number is computed.","marker":"Murray et al., 2011a"},{"why":"Defines the cause-specific mortality fraction (CSMF) accuracy formula used to evaluate population-level estimates.","marker":"Murray et al., 2011b"},{"why":"Provides the LCVA Bayesian baseline and the leave-one-site-out protocol that the GPT-5 comparison must beat.","marker":"Li et al., 2024"}],"fun_headline_variants":["GPT-5 lifts verbal autopsy accuracy by 5–10 points","LLMs beat classic verbal autopsy algorithms","Language models sharpen cause-of-death predictions","AI-assisted verbal autopsy gains 5–10% over baseline","GPT-5 reads death narratives, improves cause-of-death accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes a fair comparison: the LLM sees the free-text narrative while the traditional baseline sees only structured symptoms, so the 5–10% gap may be an input-modality effect rather than better diagnostic reasoning.","fun_headline_variants_meta":{"raw":{"variants":["GPT-5 lifts verbal autopsy accuracy by 5–10 points","LLMs beat classic verbal autopsy algorithms","Language models sharpen cause-of-death predictions","AI-assisted verbal autopsy gains 5–10% over baseline","GPT-5 reads death narratives, improves cause-of-death accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1589,"prompt_tokens":926,"completion_tokens":663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":542,"tokens_out":663,"duration_ms":5959,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:58:34.236286+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GPT-5 on the same verbal autopsy records with the free-text narrative removed, keeping only structured symptom answers and demographics; if its accuracy falls to LCVA's level or below, the reported 5–10% gain comes from reading narratives, not from reasoning over symptoms.","supporting_citations":[],"review_version":1}