{"id":"22c79595-f2c3-4825-a97f-06f3baeff97a","arxiv_id":"2510.10065","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Imputation accuracy over left-out question blocks can identify and roughly correct wrong cause-of-death probability tables in verbal autopsies using only unlabeled questionnaires, under a block-conditional-independence assumption.","lead":"A statistics working paper proposes a way to check and partly correct the 'probbase' probability tables used in verbal autopsies, using only unlabeled interview data. If it works, it could reduce the need for expensive physician-coded training data when updating cause-of-death algorithms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central identification claim (Eq. 11) depends on Theorem 2's stipulated 'if and only if' calibration condition; no uniqueness is shown for the actual VA algorithm, so I(q̂) minima may be spurious.","rationale":"The reader identifies Assumption 1 (block-conditional independence) as the weakest assumption, with Theorem 2's strong 'if and only if' condition also flagged. I agree that block misspecification is a serious practical threat. However, my most load-bearing concern is more specific: even granting Assumption 1 and correct blocks, the central claim of unique recovery is not established for the actual VA algorithm. Theorem 1 supplies only minimality of the true probbase; Theorem 2, the only uniqueness result, assumes exactly the kind of perfect calibration that the evaluation algorithm (InterVA4) is admitted to lack. The paper's simulations are local and do not test for competing global minima. This is a correctness risk to the headline claim, not just a robustness caveat. It does not, however, make the paper worthless: the working-paper framing, the clean Theorem 1, and the local-gradient simulations give some empirical support for a more modest claim about detecting nearby perturbations. I would therefore keep the reader's CONDITIONAL verdict rather than escalate to REJECT. My agreement is partial because the reader's primary emphasis on block misspecification is a related but distinct concern; the uniqueness/calibration gap is the sharper point against Eq. (11).","tokens_in":20441,"tokens_out":3796,"duration_ms":36240,"concrete_test":"Construct a small model satisfying Assumption 1 (e.g., s=4 questions in two blocks, r=2 causes, fixed true q and prior π). Use InterVA4 (or a deliberately miscalibrated posterior that ignores one block) as V. Compute I(q̂) exactly or with large Monte Carlo on a dense grid of q̂, and check whether any q̂ ≠ q has I(q̂) ≤ I(q). If such a spurious minimum exists, Eq. (11) fails; if none exists over multiple settings, uniqueness is supported despite the violated calibration condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (11) asserts q*_A := arg min I_A(q̂) ≈ arg min I(q̂) ≈ q. Theorem 1 only establishes the one-way statement that the true probbase is a minimizer: for any V' and q̂, I^{V'}(q̂) ≥ I^{V0}(q). It does not establish uniqueness. The uniqueness guarantee is Theorem 2, whose key premise is an 'if and only if' condition: the VA posterior is exactly correct on held-out blocks iff the remaining probbase entries are correct. This is a stipulated identifiability condition, not derived from Assumption 1 or from any standard VA algorithm. The authors themselves note that InterVA4 violates the weaker complete-calibration assumption (Assumption 2), so in the very setting used for evaluation, Theorem 2's condition is not known to hold. Consequently, the central claim that minimizing imputation cross-entropy recovers the true probbase has no theoretical support for the implemented method: incorrect probbases may also be global minima. The simulations do not close this gap. They do not search for global minima of I_A; they compare four hand-constructed probbases and compute local gradient signs starting from a probbase already close to q. A competing minimum far from q would not be detected by this procedure. Thus the central claim is under-supported exactly where it matters for the method's practical use.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method to calibrate or verify the 'probbase' matrix used in verbal autopsy algorithms using only unlabelled questionnaires. The idea is to quantify imputation accuracy I_A(q̂): mask all answers in a conditionally independent block, compute a posterior over causes from the remaining answers using a candidate probbase, and score the predicted answer probabilities by cross-entropy. Theorems 1 and 2 claim that the true probbase minimizes this objective (Theorem 1) and, under a stipulated 'if and only if' calibration condition, is the unique minimizer (Theorem 2). Eq. (11) asserts that minimizing I_A recovers the true probbase. Simulations on data generated under the block-conditional independence assumption with the InterVA4 algorithm show that coarsely incorrect probbases have worse imputation accuracy and that local gradient signs can detect perturbed entries with AUC ≈ 0.75–0.84. The paper is explicitly a working paper: it lists many open theoretical and practical questions and warns against real-world use.","tokens_in":20836,"tokens_out":4432,"duration_ms":38457,"significance":"If the central identification claim held, the method would allow VA probbases to be checked or partially corrected without gold-standard cause-of-death labels, which is valuable for the WHO 2022 questionnaire update and for hard-to-reach populations. The paper is transparent: code is public, assumptions are stated, and limitations are acknowledged explicitly. However, the uniqueness part of the identification claim is not established for the implemented algorithm; it rests on an assumed identifiability condition, and the simulations do not rule out spurious global minima. The significance is therefore real but conditional on further theoretical and numerical work.","major_comments":[{"comment":"The central identification claim q*_A := arg min I_A(q̂) ≈ q is supported only by Theorem 2, whose key premise is a stipulated 'if and only if' condition: V(α_-ℓ, q̂) = P(D|A_-ℓ) for all α iff q̂_{jk} = q_{jk} for all k ∉ B_ℓ. This is an identifiability assumption, not a consequence of Assumption 1 or of any standard VA algorithm. As the authors note, InterVA4 violates the weaker complete-calibration Assumption 2, so for the algorithm used in the simulations Theorem 2's condition is not known to hold. Consequently, Theorem 1 (one-way: the true probbase is a minimizer) plus Theorem 2 under an assumed condition does not establish Eq. (11) for the implemented method; incorrect probbases may also be global minima.","section":"§2.4, Theorem 2 and Eq. (11)"},{"comment":"The simulations do not test for the existence of spurious global minima. The four probbases compared in Table 1 are hand-constructed perturbations, and the gradient analysis in Eq. (12) only evaluates one-step local changes from a starting probbase already close to q. A competing probbase far from q that also minimizes I_A would not be detected by this procedure. To support Eq. (11) empirically, the authors should report minimization runs from multiple random starting points, including chaotic probbases, or otherwise demonstrate that no other local minima are encountered. As it stands, the AUC ≈ 0.75 result supports local detection of perturbations but not the global recovery claim.","section":"§3.2–3.3"},{"comment":"The block partition {B_ℓ} is learned from the same RandomPhysician dataset used to simulate the evaluation data, and the simulation then generates data exactly under Assumption 1 with those blocks. This means the evaluation never faces block misspecification, despite the paper's own list of open questions acknowledging that correct block specification cannot be guaranteed. Since the practical usefulness of the method depends on robustness to this, and the abstract claims 'necessary and sufficient' conditions, at least one misspecification experiment (e.g., using wrong block boundaries or a block with residual dependence) is needed before the practical claim is supported.","section":"§3.1 and §4.2(4)"}],"minor_comments":[{"comment":"The abstract states that 'three conditional independence blocks of size at least the number of latent categories are necessary and sufficient for calibration.' This overstates the results: Theorem 2 gives a sufficient condition under a stipulated 'if and only if' calibration assumption, and Conjecture 1 remains open. Please temper the abstract to match the proven content.","section":"Abstract"},{"comment":"In the final displayed line of Eq. (9), the subscript in P(D=d_j | A_{-B_1}=a_{-B_1}) should be A_{-B_ℓ}, not A_{-B_1}, since the block is arbitrary. This typo obscures the derivation.","section":"§2.4, Eq. (9)"},{"comment":"The text says 'When it is slightly perturbed (I_A(hat q_3))' but the preceding definition and Table 1 use hat q_4 for the slightly perturbed probbase. Please correct the reference.","section":"§3.3, Table 1"},{"comment":"The simulation description writes '(A_k | D = d_k)' but the index should be d_j, consistent with the probbase notation. This typo appears in the sentence defining the simulation of non-demographic answers.","section":"§3.1"},{"comment":"Typo: 'a known partition of of [s]' should read 'a known partition of [s]'.","section":"§1.2"},{"comment":"The proof says 'Suppose |{B_ι}| = b_ι' but the intended quantity appears to be |B_ι|. Please clarify the notation. Also, the 'almost surely' qualifier in the theorem statement is unusual; a brief explanation of the probability space would help.","section":"Theorem 2 proof"}],"recommendation":"major_revision","confidential_remarks":"The paper is openly positioned as a working paper and a 'statement of intent'. If the journal publishes preliminary methodological reports, this framing is acceptable. However, the central identification claim needs strengthening before acceptance in a regular venue: either prove uniqueness under weaker, more defensible conditions, or substantially expand the simulations to search for competing global minima and to test block misspecification. The current gap between the abstract's 'necessary and sufficient' language and the proven content is too large to ignore."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note is right, and it matters. The central identification claim — that the probbase minimizing imputation cross-entropy is the true probbase — is supported only in one direction. Theorem 1 says the true probbase is a minimizer; Theorem 2 adds uniqueness under an if-and-only-if calibration condition that is assumed, not derived. And the simulations use InterVA4, which the authors themselves say violates the weaker calibration assumption, so Theorem 2 doesn't apply in the evaluation setting. The stress-test also correctly notes that the simulation never searches for global minima: it compares hand-picked probbases and computes local gradient signs from a probbase already near the truth. Eq. (11) is therefore an aspiration, not a proven result.\n\nBut there is real substance here. The imputation-accuracy objective (Eq. 5) and the derivative-based sensitivity predictor (Eq. 12) are new relative to the VA literature, and the practical motivation is strong: after the 2022 WHO questionnaire update, there is a validated-probbase gap, and unlabeled questionnaires are the obvious source of information. The paper is also refreshingly candid — it explicitly says the method is not ready for real-world use, labels the uniqueness claim a conjecture, and lists open problems. Lemma 1 is straightforward but correct, and Theorem 1, given Assumption 1, is sound.\n\nThe soft spots, in proportion: (1) The abstract's 'necessary and sufficient' language overstates what is shown — it is a sufficiency result under a strong identifiability condition, plus an open conjecture. (2) Assumption 1 — the block-conditional independence — is not stress-tested: blocks are learned from the same data used for evaluation, and misspecification is never examined. That matters because the method's usefulness depends on the user getting the block structure right. (3) The empirical evidence is thin: no confidence intervals on the AUCs, one simulated data generator, and the ~0.75 AUC is moderate, though perfect specificity at extreme gradient values is genuinely encouraging.\n\nWho should read this: people working on verbal autopsy methods, and statisticians interested in unsupervised calibration from imputation objectives. As a working paper it is a good preregistration-style statement of intent. With an honest abstract and either a proof or a clear label for the identifiability condition, it becomes a solid conference/journal submission.\n\nA serious referee should see it. My recommendation: engage with it — the idea deserves further work, and the authors are clearly thinking about the right problems.","headline":"Worth engaging — the imputation-accuracy idea is new and the paper is honest about its limits, but the abstract over-claims: uniqueness rests on a stipulated condition, not a derived one.","tokens_in":21297,"tokens_out":3539,"would_cite":false,"duration_ms":29881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62P10","62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Minimizing imputation error recovers the true cause-of-death probability table from unlabeled verbal autopsy questionnaires.","keywords":["verbal autopsy","probbase calibration","imputation accuracy","unlabelled data","block-conditional independence","cause-of-death assignment","cross-entropy","identifiability"],"falsifier":"Construct or observe a dataset where the block-conditional-independence assumption provably holds but two distinct probbases give identical imputation scores for all inputs, contradicting Theorem 2's uniqueness; or, on a real labelled VA dataset, perturb the probbase and check whether imputation accuracy is reliably higher for the true probbase than for the perturbed one — if not, the minimality claim is empirically false.","tokens_in":20332,"feed_emoji":"🩺","tokens_out":3762,"duration_ms":33520,"temperature":0.7,"pith_summary":"This paper tries to establish that a verbal autopsy 'probbase' — the table of answer probabilities per cause of death — can be checked and partially corrected using only completed questionnaires, without any cause-of-death labels. The central idea is to measure how well a candidate probbase imputes answers that were artificially set to missing; the paper argues that the true probbase minimizes this imputation loss, so the probbase that best imputes answers is the right one. If true, this would let global health programmes validate and update probbases when physician-coded training data is unavailable, as after the 2022 revision of the standard verbal autopsy questionnaire. The paper also shows that under a block-conditional-independence assumption, three blocks of questions of size at least the number of causes are essentially necessary and sufficient for identification. The authors present this as a preliminary working paper, with simulations suggesting perturbed probbase entries can be flagged with about 75% correctness using 1500 questionnaires.","feed_headline":"Imputation loss finds the true cause-of-death table","feed_subtitle":"A verbal autopsy probbase can be checked and partly corrected using only questionnaires, no labels needed.","key_machinery":"The key mechanism is the imputation cross-entropy I(qhat): delete one block of answers, compute the posterior over causes from the remaining answers using a VA algorithm and the candidate probbase, predict the deleted answer via F_k(pi,qhat) = sum_j pi_j qhat_jk, and score with cross-entropy. The expansion in Eq. (9) shows that under block-conditional independence (Assumption 1), the imputed probability equals the true conditional probability exactly when qhat = q. This identity carries the argument: it converts an unlabelled-data objective into a function whose minimum is the object of inference. Theorem 2's uniqueness rests on the strong condition that a VA algorithm returns the true poste","core_discovery":"The paper's central claim is Eq. (11): the probbase q*_A that minimizes the imputation cross-entropy I_A(qhat) is approximately the true probbase q. Concretely, if one deletes all answers in a block of questions, uses the candidate probbase to compute a posterior over causes from the remaining answers, predicts the deleted answers, and measures cross-entropy against the real ones, this score is minimized when the probbase is correct. Theorem 1 proves that no other probbase and algorithm combination can beat the correct probbase under block-conditional independence, provided the VA algorithm returns the true posterior when the probbase is correct. Theorem 2 adds a strong 'if and only if' cali","pith_inferences":["The approach is a form of self-supervised learning; a natural extension is to use the imputation objective as a training loss to re-estimate a full probbase from scratch in large unlabelled corpora, a large-data limit the paper mentions but does not test.","The block-conditional-independence assumption is learned from the same simulated data used for evaluation; practical value likely hinges on whether stable question blocks transfer across populations, which is testable with multi-site datasets.","The simulation shows high-magnitude gradient values flag perturbed entries almost perfectly, so a two-stage procedure (flag then verify only flagged entries) could substantially cut the cost of probbase re-estimation.","If the uniqueness conjecture fails, the method still works as a veto: it can tell you a probbase is wrong, but not certify that a probbase is right."],"forward_implications":["Probbase verification without labels: VA programmes could check a probbase against a corpus of completed questionnaires, reducing reliance on physician-coded training data.","Partial correction: the gradient of imputation accuracy with respect to probbase entries gives a signal for whether an entry is too high or too low, so entries can be prioritised for expert review.","Algorithm assessment: with a trusted probbase, the same imputation score can compare VA algorithms, since well-calibrated algorithms give better imputation.","Adaptation to new questionnaires and populations: probbases differ across populations and change with questionnaire revisions; unlabelled calibration offers a way to update them where labelled data is scarce.","Formal identifiability: three conditionally independent blocks of size at least the number of causes are claimed to be necessary and sufficient for calibration, delineating when the approach can work."],"fun_headline_variants":["Imputation loss exposes wrong cause-of-death tables","Delete answers, predict them, calibrate autopsies","Minimizing imputation error finds true death causes","Unlabeled verbal autopsies: imputation loss corrects probbase","Cause-of-death calibration via imputation trick"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole method rests on Assumption 1: that the interview questions can be partitioned into blocks such that answers in different blocks are conditionally independent given the true cause of death; the paper learns those blocks from the very data it evaluates on, so misspecification is never tested.","fun_headline_variants_meta":{"raw":{"variants":["Imputation loss exposes wrong cause-of-death tables","Delete answers, predict them, calibrate autopsies","Minimizing imputation error finds true death causes","Unlabeled verbal autopsies: imputation loss corrects probbase","Cause-of-death calibration via imputation trick"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1023,"prompt_tokens":783,"completion_tokens":240,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":176}},"tokens_in":527,"tokens_out":240,"duration_ms":3354,"temperature":1.0,"reasoning_tokens":176,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:20:09.013018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or observe a dataset where the block-conditional-independence assumption provably holds but two distinct probbases give identical imputation scores for all inputs, contradicting Theorem 2's uniqueness; or, on a real labelled VA dataset, perturb the probbase and check whether imputation accuracy is reliably higher for the true probbase than for the perturbed one — if not, the minimality claim is empirically false.","supporting_citations":[],"review_version":1}