REVIEW 2 major objections 5 minor 33 references
Pitfalls of Administrative Censoring in Survival Models with Time-Indexed Inputs
T0 review · 2 major / 5 minor · reviewed 2026-07-14 · grok-4.5
Pith's one-line read When clinical inputs carry acquisition-date clues, a fixed study end date lets survival models predict follow-up length instead of true event risk.
desk verdict Solid structural diagnosis of a real evaluation trap in long-horizon clinical survival work; the n-year follow-up design rule is the part that will stick. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Administrative-cutoff leakage: the pathway from recoverable reference time in the input (D→X) through administrative censoring (potential follow-up A_i = τ − D_i, so D→C) into the observed survival outcome, so predicted risk tracks the observation window rather than true event time T.
What would settle it
On a multi-year clinical cohort, ensure every record has at least n years of potential follow-up after the latest input date, retrain the same survival model, and check whether fixed-horizon AUC and outcome-conditional date concordance stay above chance while a pure date-only baseline is at chance; if they do, the leakage mechanism is not driving performance.
Extended reading notes
Core claim
Administrative-cutoff leakage arises when reference time is recoverable from the time-indexed input and a shared administrative cutoff makes potential follow-up depend on that reference time. Models can then assign higher risk to more recent records simply because those records cannot be confirmed event-free over long horizons, inflating fixed-horizon AUC and—when documented event-free follow-up is itself date-dependent—Harrell’s C-index, even when the event process carries no biological information. Controlled simulations across model classes and a mammography cohort where follow-up extension removes the date-only signal support the mechanism. The resulting design principle is that for an n
Load-bearing premise
The practical claim rests on reference time being strongly recoverable from real clinical inputs and on date-linked features not being inseparable from true risk, so that redesigning follow-up alone is enough to remove the bias.
Editorial extensions
If this is right
- For any n-year clinical risk task, the cohort must guarantee at least n years of potential follow-up after the latest input date, or evaluation must be limited to shorter observable horizons.
- Fixed-horizon AUC can look strong when the model has learned only calendar position; C-index inflation requires asymmetric documented event-free follow-up or true event drift.
- Date-recoverability probes, administrative truncation rate, date-only metric benchmarks, and outcome-conditional date concordance can flag the failure before deployment.
- IPCW reweighting does not create missing long-horizon event-free controls for recent records; extending follow-up or restricting the cohort is required.
- Balancing observed attributes is insufficient because the bias is induced by label construction under a fixed cutoff, not by sampled cohort composition alone.
Reading between the lines
- The same geometry likely affects any multi-year prediction task that uses acquisition-dated signals—radiology, wearables, longitudinal labs—whenever registries or extraction pipelines impose a hard end date.
- Published n-year AUCs on multi-year imaging cohorts may systematically overstate risk discrimination if they do not report potential follow-up after the latest exam.
- When extra follow-up is impossible, representation-level date debiasing could help, but residual date signal entangled with true risk would remain a hard limit.
- Clinical AI reporting checklists could require zero administrative truncation for the claimed horizon, or disclosure of date-only benchmarks, before n-year discrimination is asserted.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper identifies administrative-cutoff leakage as a structural failure mode in survival models that use time-indexed inputs (images, EHR snapshots, labs) collected over many calendar years. When outcomes are observed only up to a shared administrative cutoff τ, potential follow-up A_i = τ − D_i is shorter for more recent records. If reference time D is recoverable from the input X, a model can learn to predict observation window rather than true event risk. The authors formalize this via causal graphs that separate non-informative censoring, classical informative censoring, two administrative-censoring mechanisms (symmetric and asymmetric documented event-free follow-up), and genuine event drift (D o T). They propose dataset-level diagnostics (date-recoverability probe, ρ_admin(n), reference-time-only AUC/C) and a model-level Outcome-Conditional Date Concordance (OCDC) statistic, then show in controlled simulations (N=10k, ρ sweep, multitask/discrete-time/Cox) that S1 inflates fixed-horizon AUC but not Harrell’s C-index, while S2 inflates both. A same-image mammography experiment (V1 original cutoff vs V2 extended follow-up) reproduces the pattern and shows that ensuring at least n years of potential follow-up after the latest input date removes the date-only signal. The central design principle follows directly: for an n-year task the cohort must satisfy τ − max D_i ≥ n.
Significance. If the mechanism holds, the paper supplies a concrete, checkable failure mode that is distinct from classical informative censoring and from ordinary dataset-composition shortcuts. The causal separation (Fig. 1), the independent diagnostics (date probe trained without survival labels; reference-time-only metrics; OCDC after outcome matching), the S1/S2/S3 simulation design, and the same-image V1 o V2 follow-up extension are all falsifiable and reproducible contributions. The resulting cohort-design rule—require at least n years of potential follow-up after the latest input date—is immediately actionable for clinical risk-model pipelines and for dataset construction. Even if severity varies by modality, documenting when and how administrative cutoffs can inflate AUC and C-index is a useful addition to the survival-ML literature.
major comments (2)
- §7.1–7.3 and Table 4: the real-world claim rests on a single internal FFDM cohort in which a year probe reaches AUC 0.83 and follow-up extension (V1 o V2) removes date-only signal. The paper does not report whether residual date-linked image features remain entangled with true cancer risk after extension, nor does it test other modalities (EHR, labs, CT) where date recoverability may be weaker or more biologically entangled. Without at least one additional modality or an external mammography cohort, the practical severity and the sufficiency of follow-up extension alone remain under-supported relative to the breadth of the abstract claim.
- §6.3 / Table 3: under S2, restricting the cohort to records with D_i ≤ τ−5 reduces but does not eliminate AUC@5 inflation (0.806 o 0.630) while full follow-up extension returns performance to chance. The manuscript correctly notes this, yet the abstract and §8 design principle present “at least n years of potential follow-up” as the primary safeguard without quantifying residual bias under realistic incomplete documentation. A short sensitivity analysis or explicit caveat that restriction alone is incomplete under asymmetric event-free follow-up would strengthen the load-bearing recommendation.
minor comments (5)
- Figure 1 caption and surrounding text: “administrative cenrsoring” is misspelled; also “patients enters the study later” (§1) and “for ann-year” (Abstract) need correction.
- §4.2: the OCDC tolerance w is introduced but never given a concrete value or sensitivity check in the simulation or mammography experiments; stating the chosen w (or that results are stable across a range) would aid reproducibility.
- Table 1 panel (b) lists C(D) only once while AUC_n(D) is shown per horizon; clarifying that C(D) is horizon-agnostic (or reporting it per horizon if computed that way) would avoid ambiguity.
- §5 briefly mentions representation-level remedies (harmonization, adversarial training) but evaluates only cohort-design fixes; a sentence on why those were not tested, or a pointer to future work, would balance the mitigation discussion.
- References: a few arXiv preprints (e.g., Kvamme & Borgan 2019) could be updated to published versions if available; otherwise the citation list is adequate.
Circularity Check
No circularity: the leakage claim is a structural mechanism demonstrated by controlled construction and independent diagnostics, not a result forced by definition or self-citation.
full rationale
The paper's central derivation is: if reference time D is recoverable from time-indexed inputs X and potential follow-up is A_i = τ − D_i under a shared administrative cutoff, then a model that recovers D can discriminate on observation window rather than event risk T. That implication is not self-definitional. In simulations, T is generated independent of X and D (S1/S2), date signal is injected via controlled ρ, and above-chance AUC/C-index is therefore a consequence of the observation process, not a fitted target renamed as prediction. Dataset diagnostics (date probe trained without survival labels; reference-time-only AUC_n(D) and C(D); ρ_admin(n)) and model-level OCDC (outcome-matched pairs) are independent handles, not tautologies of the survival loss. S3 is explicitly a positive control so residual discrimination after mitigation is not rebranded as leakage. Cohort remedies (extend τ or restrict to D_i ≤ τ − n) follow from making the horizon observable, not from circular redefinition of the metric. The mammography V1→V2 experiment holds images fixed and only changes documented follow-up, so the drop in date-only signal and OCDC is an empirical contrast, not a fit-to-target. Citations are standard survival and shortcut-learning literature; no load-bearing uniqueness theorem or ansatz is imported from the present authors. No step reduces Eq. X to Eq. Y by construction or renames a fitted parameter as a first-principles prediction. Score 0 is appropriate.
Assumptions & free parameters
free parameters (5)
- feature–date correlation ρ
- event hazard λ=0.07
- event-drift coefficient β=1.5
- loss-to-follow-up mean (10 years) and asymmetric documented follow-up rules in S2
- OCDC tolerance w and horizon set {1…5}
assumptions (5)
- domain assumption Observed survival data are (Y=min(T,C), δ=1{T≤C}) with potential administrative bound C ≤ τ−D for shared cutoff τ.
- domain assumption Conditional independence C ⊥ T | X is the usual non-informative censoring assumption; classical informative censoring is a latent U violation of that independence.
- domain assumption Time-indexed clinical inputs can contain recoverable calendar-time signatures (devices, protocols, documentation, practice).
- domain assumption Horizon-n event-free status is only confirmable when potential follow-up A_i ≥ n (or via documented last event-free time L_i under asymmetric ascertainment).
- standard math Standard survival estimators (Cox PH, discrete-time survival, fixed-horizon AUC, Harrell C) and IPCW behave as in the cited literature.
invented entities (2)
-
administrative-cutoff leakage
independent evidence
-
Outcome-Conditional Date Concordance (OCDC)
Cite this review
Pith. "Pith review of Pitfalls of Administrative Censoring in Survival Models with Time-Indexed Inputs." pith.science (2026). https://pith.science/paper/HVHVAUCD
@misc{pith2026260710466,
author = {Pith},
title = {Pith review of: Pitfalls of Administrative Censoring in Survival Models with Time-Indexed Inputs},
year = {2026},
howpublished = {\url{https://pith.science/paper/HVHVAUCD}},
note = {Machine review of arXiv:2607.10466}
}
read the original abstract
Survival models can model time-to-event outcomes using partially observed data. They are widely used in clinical prediction, including cancer risk, disease progression, treatment response, and mortality. Recent models often rely on rich inputs collected at a specific clinical encounter, such as medical images, laboratory tests, electronic health record snapshots, or sensor measurements. In large retrospective datasets, these inputs are usually collected over many calendar years. As a result, they may contain clues about when they were acquired through changes in devices, protocols, documentation, patient mix, or clinical practice. This creates a potential failure mode when outcomes are observed only up to a fixed study end date. More recent records necessarily have less potential follow-up than older records. A model that can infer the record date from the input may therefore learn to predict how much follow-up was available rather than the patient's true risk of experiencing the event. We call this failure mode administrative-cutoff leakage. In this paper, we characterize when this leakage can occur, distinguish it from classical informative censoring and genuine temporal changes in risk, and propose practical ways to detect it. In simulations, we show that administrative-cutoff leakage can inflate fixed-horizon AUC and can also affect Harrell's C-index under realistic follow-up patterns. We then demonstrate the same behavior in a real mammography cohort. These results motivate a simple design principle for survival prediction: for an n-year prediction task, the dataset should provide at least n years of potential follow-up after the latest input date. Otherwise, the models may be subject to bias induced by administrative-cutoff leakage.
Figures
Reference graph
Works this paper leans on
-
[1]
Paul D. Allison. Discrete-time methods for the analysis of event histories.Sociological Methodology, 13:61–98, 1982
1982
-
[2]
Badgeley, John R
Marcus A. Badgeley, John R. Zech, Luke Oakden-Rayner, et al. Deep learning predicts hip fracture using confounding patient and healthcare variables.NPJ Digital Medicine, 2(1):31, 2019. 11
2019
-
[3]
In- telligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. In- telligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1721–1730, 2015
2015
-
[4]
David R. Cox. Regression models and life-tables.Journal of the Royal Statistical Society: Series B (Methodological), 34(2):187–202, 1972
1972
-
[5]
DeGrave, Joseph D
Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal.Nature Machine Intelligence, 3:610–619, 2021
2021
-
[6]
Sheline, et al
Jean-Philippe Fortin, Nicholas Cullen, Yvette I. Sheline, et al. Harmonization of cortical thickness measurements across scanners and sites.NeuroImage, 167:104–120, 2018
2018
-
[7]
Wichmann
Robert Geirhos, J¨ orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020
2020
-
[8]
Gensheimer and Balasubramanian Narasimhan
Michael F. Gensheimer and Balasubramanian Narasimhan. A scalable discrete-time survival model for neural networks.PeerJ, 7:e6257, 2019
2019
Show all 33 references
-
[9]
Poynton, Shyam Visweswaran, and Kayhan Batmanghelich
Shantanu Ghosh, Clare B. Poynton, Shyam Visweswaran, and Kayhan Batmanghelich. Mammo- CLIP: A vision language foundation model to enhance data efficiency and robustness in mam- mography. InMedical Image Computing and Computer Assisted Intervention – MICCAI 2024, Lecture Notes ...
2024
-
[10]
Ai recognition of patient race in medical imaging: a modelling study.The Lancet Digital Health, 4(6):e406–e414, 2022
Judy Wawira Gichoya, Imon Banerjee, Ananth Reddy Bhimireddy, et al. Ai recognition of patient race in medical imaging: a modelling study.The Lancet Digital Health, 4(6):e406–e414, 2022
2022
-
[11]
Harrell, Kerry L
Frank E. Harrell, Kerry L. Lee, and Daniel B. Mark. Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15(4):361–387, 1996
1996
-
[12]
Heagerty and Yingye Zheng
Patrick J. Heagerty and Yingye Zheng. Survival model predictive accuracy and ROC curves. Biometrics, 61(1):92–105, 2005
2005
-
[13]
Sjoding, and Jenna Wiens
Sarah Jabbour, David Fouhey, Ella Kazerooni, Michael W. Sjoding, and Jenna Wiens. Deep learning applied to chest x-rays: Exploiting and preventing shortcuts. InProceedings of the Machine Learning for Healthcare Conference (MLHC), volume 126, pages 1–32, 2020
2020
-
[14]
Ji, Ahmed M
Christina X. Ji, Ahmed M. Alaa, and David Sontag. Large-scale study of temporal shift in health insurance claims.arXiv preprint arXiv:2305.05087, 2023
2023 arXiv
-
[15]
Kalbfleisch and Ross L
John D. Kalbfleisch and Ross L. Prentice.The Statistical Analysis of Failure Time Data. Wiley, Hoboken, NJ, 2 edition, 2002
2002
-
[16]
Kaplan and Paul Meier
Edward L. Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53(282):457–481, 1958
1958
-
[17]
Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger
Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network.BMC Medical Research Methodology, 18(1):24, 2018
2018
-
[18]
Seong Ho Kim et al. Deep-learning-based natural language processing of serial free-text ra- diological reports for predicting rectal cancer patient survival.Korean Journal of Radiology, 22(12):1993–2004, 2021
1993
-
[19]
Continuous and discrete-time survival prediction with neural networks.arXiv preprint arXiv:1910.06724, 2019
H˚ avard Kvamme and Ørnulf Borgan. Continuous and discrete-time survival prediction with neural networks.arXiv preprint arXiv:1910.06724, 2019
1910 arXiv
-
[20]
Larrazabal, Nicol´ as Nieto, Victoria Peterson, Diego H
Agostina J. Larrazabal, Nicol´ as Nieto, Victoria Peterson, Diego H. Milone, and Enzo Ferrante. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis.Proceedings of the National Academy of Sciences, 117(23):12592–12594, 2020. 12
2020
-
[21]
Deep learning.Nature, 521(7553):436–444, 2015
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.Nature, 521(7553):436–444, 2015
2015
-
[22]
Mikhael, Jeremy Wohlwend, Adam Yala, Ludvig Karstens, Justin Xiang, Angelo K
Peter G. Mikhael, Jeremy Wohlwend, Adam Yala, Ludvig Karstens, Justin Xiang, Angelo K. Takigami, Patrick P. Bourgouin, Puiyi Pang, Daniel Su, Lecia V. Sequist, Sridhar Sangha, et al. Sybil: A validated deep learning model to predict future lung cancer risk from a single low-do...
2023
-
[23]
Dissecting racial bias in an algorithm used to manage the health of populations.Science, 366(6464):447–453, 2019
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations.Science, 366(6464):447–453, 2019
2019
-
[24]
Shortcut learning in medical ai hinders generalization.NPJ Digital Medicine, 2024
Christopher Ong Ly et al. Shortcut learning in medical ai hinders generalization.NPJ Digital Medicine, 2024
2024
-
[25]
Lawrence, edi- tors.Dataset Shift in Machine Learning
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, edi- tors.Dataset Shift in Machine Learning. MIT Press, Cambridge, MA, 2009
2009
-
[26]
Robins and Dianne M
James M. Robins and Dianne M. Finkelstein. Correcting for noncompliance and dependent cen- soring in an AIDS clinical trial with inverse probability of censoring weighted (IPCW) log-rank tests.Biometrics, 56(3):779–788, 2000
2000
-
[27]
Nielsen, Andreas P
Hans-Christian Thorsen-Meyer, Anders B. Nielsen, Andreas P. Nielsen, Benjamin S. Kaas-Hansen, Peter Toft, Jens Schierbeck, Thomas Strøm, Piotr J. Chmura, Malte Heimann, Lukas Dybdahl, et al. Discrete-time survival analysis in the critically ill: a deep learning approach using ...
2022
-
[28]
Tsiatis.Semiparametric Theory and Missing Data
Anastasios A. Tsiatis.Semiparametric Theory and Missing Data. Springer, New York, 2006
2006
-
[29]
Food and Drug Administration
U.S. Food and Drug Administration. Clinical trial endpoints for the approval of cancer drugs and biologics: Guidance for industry, 2018
2018
-
[30]
Steiner, Zhaoyang Xu, Apaar Sadhwani, Hongwu Wang, Isabelle Flament-Auvigne, et al
Ellery Wulczyn, David F. Steiner, Zhaoyang Xu, Apaar Sadhwani, Hongwu Wang, Isabelle Flament-Auvigne, et al. Deep learning-based survival prediction for multiple cancer types us- ing histopathology images.PLOS ONE, 15(6):e0233678, 2020
2020
-
[31]
Mikhael, Fredrik Strand, Gigin Lin, Kevin Smith, Yung-Liang Wan, Leslie Lamb, Kevin Hughes, Constance Lehman, and Regina Barzilay
Adam Yala, Peter G. Mikhael, Fredrik Strand, Gigin Lin, Kevin Smith, Yung-Liang Wan, Leslie Lamb, Kevin Hughes, Constance Lehman, and Regina Barzilay. Toward robust mammography- based models for breast cancer risk.Science Translational Medicine, 13(578):eaba4373, 2021
2021
-
[32]
Zech, Marcus A
John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.PLOS Medicine, 15(11):e1002683, 2018
2018
-
[33]
Mitigating unwanted biases with adver- sarial learning
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340, 2018. 13
2018
Reviewed July 14, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.