{"id":"225728fe-5d98-4af4-a7e4-339090465d52","arxiv_id":"2505.14803","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SurvUnc trains a covariate-based meta-model on anchor-computed concordance errors to produce post-hoc uncertainty scores for any survival model.","lead":"A new framework, SurvUnc, uses a small secondary model to score how uncertain any survival-analysis model is about its predictions, without changing the original model. The scores help decide which patient predictions to trust and which to flag for review.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CCR assumption and uncensored-only evaluation leave the central claim unverified for the majority-censored real data.","rationale":"The reader's weakest_assumption identifies the same CCR assumption; I agree. This is the most load-bearing concern because it sits at the intersection of training-label validity and evaluation coverage. The meta-model's entire training set D_meta is built from uncensored samples, and Eq. (3) only makes sense for uncensored samples; the only justification for transferring those labels to censored samples is CCR. The paper's own datasets are predominantly censored, yet the evaluation selectively discards censored samples for metric computability. Without a test on censored samples, the central claim of model-agnostic uncertainty quantification for any survival sample is unverified for the majority of real data. This does not refute the method; it makes the current evidence conditional, matching the reader's CONDITIONAL verdict. I therefore recommend no verdict change. The undefined denominator in Eq. (3) is a secondary concrete flaw that reinforces the need for the conditional.","tokens_in":23937,"tokens_out":8424,"duration_ms":69549,"concrete_test":"Using SAC3, which provides ground-truth survival curves, simulate informative censoring by drawing each censoring time from a distribution that depends on covariates (for example, censor earlier for high-risk patients). Train SurvUnc with the paper's CCR assumption and protocol, then compute the Pearson correlation between SurvUnc's uncertainty scores and the true integrated squared error of the base model's survival curves on censored and uncensored test samples. If the correlation on censored samples is substantially lower than on uncensored samples, the CCR assumption is load-bearing and the current evaluation, which only uses uncensored test samples, overstates generalizability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SurvUnc quantifies uncertainty for any survival model depends on the assumption stated in Section 3.3, before Eq. (3), that censoring occurs completely at random (CCR). Under CCR, the uncensored samples used to compute anchor-based labels are a representative random subset of the population, so a meta-model trained on them can generalize to censored samples. However, the real datasets are majority-censored (FLCHAIN: 69.9% censored; SEER-BC: 73.7% censored), and Section 4.2 states that selective prediction and subsequent analyses discard or focus only on uncensored samples. If censoring is informative, for example if patients with worse prognoses are more likely to be lost to follow-up, the concordance-error labels are biased toward the uncensored subpopulation and the meta-model never observes a censored sample during training. The paper provides no evaluation on censored test samples, so its superiority claims do not cover the most common case in real survival data. A related technical gap is that Eq. (3) is undefined when no anchor has t_A_k > t_j, because the denominator is zero; the paper does not explain how such training samples are handled, which could exclude the longest-surviving patients from meta-training.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SurvUnc, a post-hoc, model-agnostic uncertainty quantification framework for survival models. The method trains a lightweight meta-model (random forest or MLP) on the base model's covariates alone, using labels derived from an anchor-based concordance-error measure: for each uncensored training sample, the label is the proportion of anchors with longer observed event times for which the base model incorrectly ranks the survival probabilities. The trained meta-model then outputs an uncertainty score for a new sample using only its covariates. The authors evaluate SurvUnc across four datasets and five survival models on three tasks: selective prediction (discarding high-uncertainty samples and recomputing C_td and IBS), misprediction detection (correlation between uncertainty scores and per-sample IBS), and out-of-domain detection (SEER-BC vs. SEER-HD). They report that SurvUnc outperforms MC-Dropout and Deep Ensemble baselines, and also outperforms the native Bayesian uncertainty of BNNSurv.","tokens_in":24195,"tokens_out":5885,"duration_ms":56453,"significance":"If the empirical claims hold, the framework would be practically useful: it is truly post-hoc and model-agnostic, requires no access to base-model parameters, and the implementation is provided. The experiments cover a reasonable spread of survival models (DeepSurv, DeepHit, DSM, RSF, BNNSurv) and datasets, and the authors report bootstrap resampling and Wilcoxon signed-rank tests. However, the paper's central claim that it quantifies uncertainty for 'any survival model' is not supported by the evaluation as currently presented: selective prediction lacks a random-discarding baseline, all main analyses are restricted to uncensored test samples, and the label definition in Eq. (3) is undefined for a non-negligible subset of training samples. These gaps are load-bearing because they directly affect the interpretation of every reported improvement. The work is a reasonable step toward standardized UQ evaluation for survival analysis, but the manuscript needs additional experiments and clarifications before the central claims can be accepted.","major_comments":[{"comment":"The selective prediction experiment lacks a random discarding baseline. The paper reports C_td after removing 10%, 30%, and 50% of test samples according to an uncertainty score, but never reports C_td for a random subset of the same size, nor the C_td at 0% discarding for each base model. Without this baseline, the large relative improvements (e.g., 32.5% on FLCHAIN for DeepSurv) cannot be attributed to the uncertainty ordering: C_td on a smaller subset can change mechanically, and the selected subset may differ in composition. Please add a random-discarding curve (or an oracle upper bound) for every model-dataset combination, and report the no-discard C_td so that the improvement is interpretable.","section":"Section 4.2, Table 2"},{"comment":"The denominator in Eq. (3), sum over k of 1[t_j < t_A_k], is zero whenever no sampled anchor has an observed event time greater than t_j. This is likely to occur for the longest-surviving uncensored training samples, since K=50 anchors are drawn at random. The paper does not state whether such samples are excluded from the meta-training set, assigned a default label, or handled in some other way. This is a real specification gap: those samples are exactly the long-term survival cases, and their exclusion or arbitrary labeling changes the meta-model's training distribution. Please define the rule explicitly and report how many training samples have zero denominator in each dataset.","section":"Section 3.3, Eq. (3)"},{"comment":"All selective prediction and misprediction detection results are computed on uncensored test samples only, as stated in Section 4.2: 'we discard only uncensored samples, and subsequent analyses also focus on uncensored ones.' On datasets with majority censoring (FLCHAIN 69.9%, SEER-BC 73.7%), this means the reported results cover only a minority of test cases, and there is no evidence that SurvUnc provides useful uncertainty scores for censored predictions. The framework's central claim is about any survival model on censored data, so this is a major omission. Please evaluate on the full test set using censoring-adjusted metrics (e.g., IPCW-weighted IBS for misprediction, and C_td computed with all comparable pairs, or a separate reported analysis for censored samples).","section":"Section 4.2, Section 4.3, Table 1"},{"comment":"The CCR (censoring completely at random) assumption is load-bearing for label construction, because anchors and labeled training instances are drawn exclusively from uncensored samples. The paper invokes the assumption without validation, but informative censoring (e.g., sicker patients lost to follow-up) would bias the concordance-error labels toward the uncensored subpopulation. The fact that the evaluation is also restricted to uncensored test samples masks this problem. Please add a sensitivity analysis: for instance, simulate a survival dataset under informative censoring and show whether SurvUnc's scores still rank censored samples correctly, or compare the covariate distributions of uncensored anchors with those of censored samples in the real datasets.","section":"Section 3.3, Section 4.4"},{"comment":"The absolute OOD detection performance is low: AUROC values are about 0.62-0.67 and AUPRC values about 0.58-0.63. While these numbers improve over baselines near 0.5, an AUROC of roughly 0.65 is generally considered a weak detector, and the text's claim of a 'significant performance improvement' and 'notable enhancement' overstates the practical capability. Please report confidence intervals for the OOD metrics and discuss whether such AUROC levels would be acceptable for the intended high-stakes deployment scenarios.","section":"Section 4.4, Table 4"}],"minor_comments":[{"comment":"The heading 'ODD detection' appears to be a typo for 'OOD detection'. Also, in the caption of Figure 5, 'SurvUnC-RF' should be 'SurvUnc-RF'.","section":"Section 4.4"},{"comment":"The entry '0701' for TabPFN at 30% discarding appears to be missing the leading '0' and decimal point; it should likely be '0.701'.","section":"Appendix B.5, Table 12"},{"comment":"In the sentence discussing BNNSurv, 'BNNSurve' appears to be a typo for 'BNNSurv'.","section":"Section 4.2"},{"comment":"The statement 'p-value<0.001 is true for all results without *' is ambiguous: it does not specify the null hypothesis or the comparison (e.g., SurvUnc-RF vs. best baseline, or vs. MC-Dropout), and it is unclear whether the test is the Wilcoxon signed-rank test across bootstrap resamples or across repeated data splits. Please clarify the statistical test procedure.","section":"Section 4.1.5 and Table 2"},{"comment":"The indicator 1[S(t_j|x_j) >= S(t_j|x_A_k)] counts ties as 'incorrectly ordered pairs', but in survival analysis a tie in predicted survival probability is not necessarily an error. Consider using a strict inequality or a tie-handling convention consistent with the C_td definition.","section":"Section 3.3, Eq. (3)"},{"comment":"The claim that SurvUnc is 'the first model-agnostic framework capable of quantifying the uncertainty of predictions in any survival model' should be moderated or supported by a more thorough comparison with existing post-hoc UQ methods for survival analysis, since several Bayesian and ensemble approaches are model-agnostic in practice.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published in the KDD '25 proceedings, but as a journal submission the central empirical claims need to be supported by the missing random-discarding baseline and the censored-sample evaluation. The undefined denominator in Eq. (3) is a correctness issue that should have been caught before publication; it strengthens the case that the method is underspecified. The framework itself is simple, reproducible, and potentially useful; the issues are addressable with additional experiments and clarifications, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on survival analysis or post-hoc UQ. The anchor-based concordance label in Eq. (3) is the real new idea: it turns the base model's ranking errors against uncensored anchors into a supervision signal, so a covariate-only meta-model can output an uncertainty score for any survival model without touching its parameters. That is a clean, practical contribution. The evaluation is wide—four datasets, five survival models, bootstrap CIs and Wilcoxon tests—and they validate their base models against published numbers. On uncensored test samples, the selective-prediction and misprediction-detection improvements over MC-Dropout and ensembles are consistent.\n\nWhere it gets shaky is the censoring story. The meta-model is trained only on uncensored samples, and the main analyses restrict evaluation to uncensored test samples. The paper says this is to reflect real scenarios where test samples have not yet been censored, but on FLCHAIN and SEER-BC the censored patients are 70–74% of the data. The CCR assumption in Section 3.3 is what lets the meta-model generalize to censored cases, yet informative censoring is common in medical data, and there is no evaluation on censored samples or sensitivity check. Eq. (3) also has an undefined denominator when no anchor has t_A > t_j, so the longest-surviving uncensored patients would have no label; the paper doesn't say how they're handled. A random discarding baseline is missing from the selective-prediction tables, which would isolate the effect of the UQ score from the mechanical effect of discarding samples. And the OOD detection AUROC around 0.63–0.67 shows the score is useful but not sharp.\n\nI wouldn't call these fatal. The core method is sound on what it actually evaluates, and the gaps are fixable: evaluate on censored samples (or at least report calibration there), add a random baseline, and clarify the denominator. The “first model-agnostic framework” claim is a little strong given other post-hoc UQ literature, but as a meta-model method it is new.\n\nWho benefits: anyone building trust layers on top of survival models in clinical or risk settings. I'd bring it to a reading group and would cite it, with a note about the uncensored-only evidence. It deserves a full peer review, but a good reviewer should push on the censored-sample evaluation.","headline":"A genuinely new meta-learning trick for survival UQ, with an evaluation that stops at uncensored samples—worth reviewing, but the central claim needs a censored-case check.","tokens_in":24708,"tokens_out":3057,"would_cite":true,"duration_ms":24886,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SurvUnc claims to be the first model-agnostic framework that attaches a predictive-uncertainty score to any survival model, trained only on covariates and anchor-based concordance labels.","keywords":["survival analysis","uncertainty quantification","meta-model","anchor-based learning","concordance","censoring","selective prediction","out-of-domain detection"],"falsifier":"Train SurvUnc on a synthetic dataset with known survival times where censoring is informative (for instance, high-risk patients are more likely to be censored), then measure the Pearson correlation between SurvUnc's scores and actual prediction error on the censored test patients; if the completely-at-random censoring assumption is load-bearing, this correlation will drop toward zero, showing the scores do not generalize to the censored cases the framework is meant to cover.","tokens_in":23724,"feed_emoji":"🩺","tokens_out":6875,"duration_ms":72955,"temperature":0.7,"pith_summary":"SurvUnc addresses a gap: survival models predict time-to-event, but almost none of them say how much to trust a given prediction. The paper claims to fix this with a post-hoc meta-model that takes only the patient covariates and outputs an uncertainty score, without touching the survival model's architecture or parameters. Because true survival curves are never observed, the authors construct training labels from a concordance principle: among two patients, the one who dies sooner should have a lower survival probability. The meta-model learns to flag cases where the base survival model gets that ordering wrong relative to a set of 'anchor' uncensored patients. Experiments across four datasets and five survival models show the scores improve selective prediction, correlate with per-patient error, and separate in-distribution from out-of-distribution patients, outperforming dropout and ensemble baselines.","feed_headline":"Uncertainty scores now attachable to any survival model","feed_subtitle":"A covariate-only meta-model learns to flag when a survival model misranks short-lived patients.","key_machinery":"The load-bearing object is the anchor-based concordance label (Eq. 3): a per-patient number between 0 and 1 that measures the fraction of worse-off anchor patients the base model ranks incorrectly. The anchors are randomly sampled uncensored patients, and the pairwise comparison is made at the shorter patient's event time. This converts the unobservable ground-truth survival curve problem into a supervised label-generation step, after which any regression-style model (the paper tests MLP and random forest) can be trained as the meta-model $U(\\cdot): \\mathbb{R}^d \\to \\mathbb{R}_{\\ge 0}$. The machinery works because it quantifies exactly what clinicians care about—correct risk ordering—rather than predictive variance, which is why the paper argues it beats dropout and ensembles.","core_discovery":"The central claim is that predictive uncertainty in survival analysis can be learned as a supervised ranking-reliability problem even though ground-truth survival curves are unobservable. For each uncensored training patient $(\\boldsymbol{x}_j, t_j)$, SurvUnc draws $K$ uncensored anchors and counts the fraction of anchors with $t_j < t_{A_k}$ for which the base model assigns survival probability $S(t_j|\\boldsymbol{x}_j) \\ge S(t_j|\\boldsymbol{x}_{A_k})$; that fraction $y^{\\mathrm{meta}}_j \\in [0,1]$ is the training label. A meta-model $U(\\boldsymbol{x})$ trained on these labels outputs the uncertainty score at test time from covariates alone. The paper claims this is the first framework that quantifies prediction uncertainty for any survival model—neural, tree-based, or Bayesian—without modifying the model or accessing its internals, and that the resulting scores outperform MC-Dropout and deep ensembles on selective prediction, misprediction detection, and out-of-domain detection across four datasets and five base models.","pith_inferences":["Because the label uses only uncensored anchors, I expect the scores to degrade as the censoring rate rises; a natural fix would be inverse-probability-of-censoring weighting of the pairwise terms.","The framework deliberately ignores the base model's output at test time, so it cannot detect a base model that silently degrades (for example, one with corrupted weights) when the covariates look normal; adding a cheap fingerprint of the base model's predictions would be a direct ablation.","The same anchor-concordance scheme could extend to competing risks by comparing cause-specific survival probabilities, and to time-varying covariates by evaluating the label at multiple event horizons—the paper lists both as future work.","A cheap validation anyone can run on the paper's own SAC3 synthetic data is to check whether the high-uncertainty samples consistently recover the longer-survival tail shown in Figure 4; if changing the anchor set flips the ordering of high- and low-uncertainty samples, the stability of the labels with respect to the anchor sample is the thing to test."],"forward_implications":["Existing survival models—including tree-based models like RSF that cannot use MC-Dropout—can be paired with a lightweight uncertainty scorer without retraining or parameter access.","Selective prediction improves: discarding the highest-uncertainty samples raises time-dependent concordance by roughly 5–40% relative to best baselines at 50% discard, depending on model and dataset, and can nearly halve the Brier score.","Misprediction detection shows SurvUnc's scores correlate positively with per-patient Brier error across all tested models and datasets, while MC-Dropout and ensembles frequently show zero or negative correlation, so the scores can triage which predictions need expert review.","On the SEER breast-cancer versus heart-disease split, SurvUnc raises out-of-domain detection AUROC by roughly 20–30% over baselines, indicating the anchor labels encode domain knowledge.","The meta-model choice is flexible: random forest, MLP, and tabular models all fit the same labels, so the anchor-label scheme rather than a specific architecture carries the effect."],"supporting_citations":[{"why":"Supplies the post-hoc meta-model uncertainty paradigm that SurvUnc adapts from classification to survival, and the misprediction/OOD evaluation protocol.","marker":"[44]"},{"why":"Defines the concordance concept from which the anchor-based training labels are built.","marker":"[17]"},{"why":"MC-Dropout, the dropout-based baseline that SurvUnc compares against and which cannot be applied to RSF.","marker":"[14]"},{"why":"Deep Ensembles, the ensemble-based baseline that SurvUnc outperforms in the reported experiments.","marker":"[29]"},{"why":"DeepHit survival model and one of the two citations on which the completely-at-random censoring assumption rests.","marker":"[30]"},{"why":"Deep Survival Machines model and the other citation for the completely-at-random censoring assumption.","marker":"[39]"},{"why":"DeepSurv, a principal neural base model whose pretrained predictions SurvUnc quantifies.","marker":"[25]"},{"why":"Random Survival Forests, the tree-based base model that motivates the model-agnostic design because it lacks dropout.","marker":"[22]"},{"why":"BNNSurv, the Bayesian survival model whose inherent uncertainty estimates are used as a comparison baseline.","marker":"[31]"}],"fun_headline_variants":["SurvUnc: uncertainty scoring for any survival model","Post-hoc uncertainty for survival, no architecture changes","Anchors teach survival models to express doubt","Ranking-based uncertainty for survival predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training labels come only from uncensored patients, and the method assumes censoring is completely at random, so the uncensored anchors are representative of the whole population; if censoring is informative, the uncertainty scores may be biased precisely for the censored patients that dominate many real datasets.","fun_headline_variants_meta":{"raw":{"variants":["SurvUnc: uncertainty scoring for any survival model","Post-hoc uncertainty for survival, no architecture changes","Anchors teach survival models to express doubt","Ranking-based uncertainty for survival predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":2061,"prompt_tokens":1008,"completion_tokens":1053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":994}},"tokens_in":624,"tokens_out":1053,"duration_ms":10277,"temperature":1.0,"reasoning_tokens":994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:29:01.263188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SurvUnc on a synthetic dataset with known survival times where censoring is informative (for instance, high-risk patients are more likely to be censored), then measure the Pearson correlation between SurvUnc's scores and actual prediction error on the censored test patients; if the completely-at-random censoring assumption is load-bearing, this correlation will drop toward zero, showing the scores do not generalize to the censored cases the framework is meant to cover.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the post-hoc meta-model uncertainty paradigm that SurvUnc adapts from classification to survival, and the misprediction/OOD evaluation protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the concordance concept from which the anchor-based training labels are built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MC-Dropout, the dropout-based baseline that SurvUnc compares against and which cannot be applied to RSF."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Deep Ensembles, the ensemble-based baseline that SurvUnc outperforms in the reported experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepHit survival model and one of the two citations on which the completely-at-random censoring assumption rests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Deep Survival Machines model and the other citation for the completely-at-random censoring assumption."},{"cited_title":"Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger","cited_arxiv_id":null,"evidence_quote":"DeepSurv, a principal neural base model whose pretrained predictions SurvUnc quantifies."},{"cited_title":"Kogalur, Eugene H","cited_arxiv_id":null,"evidence_quote":"Random Survival Forests, the tree-based base model that motivates the model-agnostic design because it lacks dropout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BNNSurv, the Bayesian survival model whose inherent uncertainty estimates are used as a comparison baseline."}],"review_version":1}