{"id":"d5cb2e73-a438-4783-9804-7355e439a41a","arxiv_id":"1908.00690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On MIMIC-III mortality and length-of-stay tasks, temporal evaluation shows raw-feature models lose up to 0.29 AUROC across the 2008 EHR switch, while expert-defined clinical concept features cut the drop to 0.06.","lead":"Using restricted year-of-care data from MIMIC-III, this study shows that clinical prediction models lose accuracy when trained on past records and tested on future ones, with the largest drop at a 2008 hospital system switch. A manually curated grouping of raw measurements into clinical concepts largely prevents this decay, suggesting that feature representation, not just model choice, determines deployability.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Clinical aggregation mitigation relies on a cross-system ItemID mapping that cannot be built before the 2008 transition, so the reported 0.06/0.03 AUROC drops are not yet shown to be achievable in a prospective deployment.","rationale":"The reader's conditional verdict is appropriate. I read the paper as making two central claims: first, that time-agnostic MIMIC-III evaluation overstates prospective performance, and second, that expert clinical aggregation is a simple and effective mitigation. The first claim is convincingly supported by the per-year AUROC curves and tables; even though the phrase 'all tested models decay' has exceptions, for example GRU-D on length of stay in Table 4, the central tendency of decay across models and representations is clear. The second claim has a real soft spot: the Clinical Aggregations mapping is built from both sides of the 2008 system change, and all temporal training regimes apply this fixed mapping to future test years. The evaluation therefore cannot distinguish the value of aggregating into stable clinical concepts from the value of knowing the future ItemID vocabulary. This is not an internal inconsistency in the experiments; it is an external-validity gap between the experiment and the deployment scenario the paper targets. The paper's conclusion acknowledges the manual-mapping limitation, but the abstract and results sections do not qualify the 0.06 and 0.03 drops as conditional on a perfect prior mapping. The proposed test would settle the issue by removing post-shift vocabulary from the mapping construction. If the drops persist under that causally valid construction, the mitigation recommendation stands; if not, the mitigation section needs to be reframed as a demonstration of what is possible with an idealized ontology rather than what a deployer can achieve before observing the new system. I do not see a more load-bearing concern: the restricted year mapping is a reproducibility limitation rather than a correctness issue, and the maximum-drop statistic is noisy but the decay pattern is replicated across models, representations, and both tasks. Therefore the reader's conditional verdict should remain unchanged.","tokens_in":15034,"tokens_out":4938,"duration_ms":51201,"concrete_test":"Re-run the Full History mortality and length-of-stay evaluations with a causally valid Clinical Aggregation mapping: for each test year t, construct the aggregation only from ItemIDs present in training years up to t-1, and for post-2008 test items such as ItemID 220045 that have no training-time mapping, leave the aggregate feature missing and apply the same imputation used elsewhere. If the maximum AUROC drop stays near 0.06 for mortality and 0.03 for length of stay, the mitigation is prospectively achievable; if it rises toward the raw-representation values of 0.29 and 0.10, the headline numbers depend on post-transition vocabulary knowledge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core decay phenomenon is well supported: Table 1 and Figure 3 show raw-representation AUROC drops concentrated at the CareVue-to-MetaVision transition under Full History. The load-bearing weakness is the mitigation claim. The Clinical Aggregations representation (Section 4.1) is defined by attaching pre-2008 CareVue ItemIDs and post-2008 MetaVision ItemIDs to the same clinical bucket, for example grouping Heart Rate ItemID 211 with ItemID 220045. This mapping uses vocabulary that a 2007-era deployer could not observe, so the evaluation protocol supplies the mitigation with exactly the information that is unavailable in a prospective deployment. The reported reductions, from 0.29 to 0.06 AUROC for mortality and from 0.10 to 0.03 for length of stay, therefore measure the value of a perfect, manually constructed cross-system ontology rather than the performance of a deployable algorithm. The paper's own conclusion states that transfer to other EHR systems would require new taxonomies developed with clinical experts, conceding that the mapping is not automatic, but the abstract's 'simple yet effective mitigation strategy' and the headline Full History numbers overstate what can be realized without future knowledge. This does not invalidate the finding that raw features are fragile across the EHR transition, but it weakens the central practical recommendation: if the mapping cannot be built before the transition, the reported mitigation numbers are not achievable in the prospective setting the paper advocates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript quantifies the effect of temporal non-stationarity on clinical prediction models trained on MIMIC-III. Using a Limited Data Use Agreement to recover the year of each admission, the authors evaluate four classifiers (logistic regression, random forest, LSTM, GRU-D) on mortality and long length-of-stay tasks, under three chronological training regimes (2001-2002, prior-year, full history) and four feature representations (raw, PCA, CUI-based, and expert-defined clinical aggregations). The main empirical claims are that raw representations show large AUROC drops when models trained on historical data are tested on future data, with the largest drops at the 2008 CareVue-to-MetaVision transition (0.29 for mortality, 0.10 for LOS with random forests), that the clinical aggregation representation reduces these drops to 0.06 and 0.03, and that year-agnostic evaluation overstates deployable performance. The authors release code and the aggregated representations.","tokens_in":15320,"tokens_out":7770,"duration_ms":72653,"significance":"The paper addresses an important gap: because MIMIC-III dates are obscured, researchers routinely report time-agnostic performance, and the deployability of those models is unknown. The temporal evaluation protocol here is clean, the comparison across models and representations is systematic, and the code and representations are released, which makes the core measurement reproducible. The finding that raw ItemID features are fragile across the 2008 system change is likely to be influential. The mitigation result is less secure: the clinical aggregation relies on a mapping from both pre- and post-transition vocabularies, so the reported 0.06/0.03 reductions are best interpreted as an oracle upper bound rather than a demonstrated prospectively achievable improvement. With the framing corrected and the overclaim about \"all models\" fixed, the paper would be a substantial empirical contribution.","major_comments":[{"comment":"The abstract states that \"all tested state-of-the-art models decay in prediction quality\" when trained on historical data and tested on future data. This is contradicted by the manuscript's own Full History results for the length-of-stay task: in Table 4, GRU-D with the Raw representation has a max AUROC drop of only 0.01, and the table note indicates that 2003 was the worst performing year with consistent improvement in subsequent years; Clinical LR and GRU-D show positive values (+0.02 and +0.06), i.e., improvement. The claim should be restricted to the mortality task, or to the specific models and representations for which decay is actually observed.","section":"Abstract; Table 4 (Appendix E)"},{"comment":"The mitigation evaluation is not prospective. Clinical Aggregations are defined by manually grouping CareVue ItemIDs with MetaVision ItemIDs (e.g., Heart Rate ItemID 211 with ItemID 220045), so the representation decoder contains knowledge of vocabulary that only became observable after the 2008 system transition. Under the Full History regime, models are trained on pre-2008 data and tested on 2008-2012 data, but the feature mapping is constructed using both sides of the transition. A deployer at the moment of transition could not have this mapping, so the reported reductions from 0.29 to 0.06 for mortality and from 0.10 to 0.03 for LOS measure the value of a perfect manually constructed cross-system ontology rather than a deployable algorithm. The conclusion acknowledges this limitation, but the abstract and Section 5.1 present clinical aggregation as a \"simple yet effective\" mitigation without this caveat. Please either evaluate a variant in which the mapping is built only from pre-2008 data or with a delayed deployment after MetaVision codes are observed, or explicitly label the current mitigation numbers as an oracle upper bound.","section":"Section 4.1 (Clinical Aggregations) and Table 1"},{"comment":"The quantification of the overstatement is confounded with training set size. The year-agnostic RF Raw AUROC of 0.82 ± 0.02 is compared with the Full History year-averaged Raw AUROC of 0.76 ± 0.13, but the year-agnostic model is trained on all 2001-2012 data, whereas a Full History model evaluated on year y is trained only on data from 2001 through y-1. The gap may reflect the larger and more recent training set as well as the unrealistic future information. To support the causal claim that date-agnostic evaluation overstates performance, compare year-agnostic models against chronological models with matched training amounts (for example, train on 2001-2002 and evaluate each future year, or report year-agnostic performance per test year while training only on prior years), or explicitly restrict the interpretation to the combined effect of these differences.","section":"Section 5.2 (Date-Agnostic Training Overstates Performance)"},{"comment":"The headline numbers are maximum single-year AUROC drops relative to 2003, and no measure of uncertainty is reported for the per-year estimates underlying the max. A single noisy 2003 estimate can therefore drive the reported \"max loss\" values. Please report confidence intervals for year-specific AUROC (e.g., bootstrap over admissions) or a statistical test for the 2008-specific drop, so that the 0.29 and 0.06 numbers can be evaluated against sampling noise.","section":"Tables 1 and 4; Section 5"}],"minor_comments":[{"comment":"There is a typo in Appendix B: \"ItfemID\" should be \"ItemID\".","section":"Appendix B"},{"comment":"Several entries in Tables 2 and 3 have formatting artifacts such as \"0 .79\" and \"0 .68\"; please fix the typesetting.","section":"Tables 2 and 3"},{"comment":"The caption is grammatically incomplete in the sentence \"the least deviant and highest performing representation has been attained for most of the models\"; please rephrase.","section":"Figure 3 caption"},{"comment":"The manuscript says it uses \"an early version\" of the MIMIC-III data extraction code by Wang et al. (2019); please specify which version was used and whether the released code is aligned with the final mimic-extract release.","section":"Section 3.2"},{"comment":"The PCA representation chooses 68 components to match the dimensionality of Clinical Aggregations; a sensitivity analysis over the number of components would help separate the effect of dimensionality from the effect of semantic grouping.","section":"Section 4.1 (PCA)"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be publishable after a major revision. The core temporal-decay finding is sound and valuable, but the abstract and conclusion overstate both the universality of the decay and the deployability of the clinical aggregation mitigation. The oracle-mapping issue is a correctness concern for the headline mitigation claim, not a reason to reject the paper, because the temporal evaluation protocol and released code are substantial contributions. I would also encourage the editor to ensure the final version reports uncertainty around the year-specific AUROC estimates that drive the maximum-drop tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is the year-resolved evaluation. They got a Limited Data Use Agreement for MIMIC-III dates and show that standard date-agnostic benchmarking overstates deployable performance: raw features drop sharply at the 2008 CareVue-to-MetaVision switch (max 0.29 AUROC mortality, 0.10 LOS on random forest under Full History). That is a real, subfield-relevant result. The measurement design is clean, the tables support the core decay phenomenon, and the authors are honest that the year mapping is restricted.\n\nWhat's also genuinely new: the Clinical Aggregate representation, grouping 181 raw ItemIDs into 68 expert-defined concepts spanning the two systems. It holds up better across the transition, and the release of code and representation is useful.\n\nNow the soft spots, in proportion.\n\nThe abstract says \"all tested state-of-the-art models decay.\" Table 4 shows GRU-D on LOS has max drop 0.01 with an asterisk noting 2003 was the worst year, so no post-2003 decay. That is a minor but real overstatement.\n\nBigger: the headline mitigation numbers—0.29 to 0.06 and 0.10 to 0.03—are RF best-case numbers, not average across models. Table 1 shows LSTM under Clinical still drops 0.13 on mortality. So \"simple yet effective mitigation\" is doing work the aggregate doesn't fully support.\n\nThe load-bearing weakness is prospective deployability of the Clinical Aggregate mapping. The mapping groups pre-2008 CareVue ItemIDs with post-2008 MetaVision ItemIDs, e.g. Heart Rate 211 and 220045. A 2007-era deployer cannot know MetaVision's vocabulary. So the 0.06/0.03 numbers measure the value of a perfect, hand-built cross-system ontology. The paper's own conclusion concedes that transfer to other EHRs requires new expert taxonomies. This doesn't invalidate the finding that raw features are fragile; it means the mitigation is an existence proof and a resource, not an algorithm you could run prospectively. The authors should frame it that way.\n\nCitation pattern is fine; prior drift work, Gong et al., Subbaswamy, Jung and Shah are all there. The self-consistency of the empirical section holds. I'd send this to a serious referee with a request to fix the overclaims and to discuss the mapping limitation up front. It deserves to be in the literature; it just needs to be more careful about what \"deployable\" means.","headline":"Solid, important empirical study of temporal drift on MIMIC-III; read the headline mitigation numbers as an upper bound from a hand-built ontology, not a deployable strategy.","tokens_in":822,"tokens_out":700,"would_cite":true,"duration_ms":21033,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hospital prediction models trained without dates lose up to 0.29 AUROC when records switch systems, and grouping measurements into clinical concepts cuts that loss to 0.06.","keywords":["temporal generalization","concept drift","clinical machine learning","electronic health records","MIMIC-III","feature representation","mortality prediction","length-of-stay prediction"],"falsifier":"Re-run the Full History experiment but derive the Clinical Aggregation mapping only from pre-2008 ItemIDs and their text descriptions, then evaluate on 2008-2012 data; if the AUROC drop approaches the raw representation's 0.29 rather than the reported 0.06 for mortality, the mitigation is not achievable prospectively. A second check: repeat the same study on another EHR dataset with a documented system transition and see whether expert-defined clinical concepts keep AUROC declines below raw-code declines.","tokens_in":14821,"feed_emoji":"🏥","tokens_out":10838,"duration_ms":99065,"temperature":0.7,"pith_summary":"Machine learning models trained on electronic health records are usually evaluated on randomly split data, ignoring when care occurred; this paper argues that such benchmarks overstate how well the models will work when deployed on future patients. Using restricted year-of-care information from the MIMIC-III critical care database, the authors show that every model they test loses predictive accuracy when trained on historical data and evaluated on future data, with the largest drop occurring at the 2008 transition from the CareVue to the MetaVision recording system. The paper's main constructive result is that a manually built representation, which groups 181 raw database codes into 68 clinically meaningful measurements, reduces the worst AUROC drop from 0.29 to 0.06 for mortality prediction and from 0.10 to 0.03 for long length-of-stay prediction. The takeaway is that deployable clinical machine learning requires time-aware evaluation and feature representations that are not tied to the changing internals of a hospital's records system.","feed_headline":"Hospital record models lose 0.29 prediction score after a 2008 switch","feed_subtitle":"The 2008 records switch cuts prediction quality by 0.29; expert feature grouping cuts the loss to 0.06.","key_machinery":"The Clinical Aggregation representation carries the argument: it replaces database-specific measurement codes (ItemIDs) with 68 manually curated clinical concepts, each defined by a mapping that explicitly bridges pre-2008 CareVue codes and post-2008 MetaVision codes for the same biophysical quantity, then converts to canonical units and averages. This makes the feature space semantically stable when the software that records measurements changes. The surrounding machinery is the temporal evaluation protocol, which trains under three year-aware regimes (train on 2001-2002 only, prior year only, or full history) and tests on each future year, plus the comparison representations: Raw, PCA, and UMLS-based CUI Code Spanning. The comparison shows that the manual semantic mapping, not automatic dimensionality reduction or ontology grouping, is what preserves performance under drift.","core_discovery":"On MIMIC-III, prediction models trained on historical data and tested on future data suffer real, measurable decay, and the raw representation commonly used in deep learning is the most fragile. At the 2008 switch from CareVue to MetaVision, a random forest trained on all prior data loses 0.29 AUROC for in-ICU mortality and 0.10 AUROC for long length-of-stay; LSTM and GRU-D models also decay sharply, and many do not recover even with years of additional training data. The paper's proposed Clinical Aggregation representation, which manually maps the 181 raw ItemIDs into 68 clinical concepts with canonical units, cuts these maximum drops to 0.06 and 0.03 and also lowers year-to-year variance. The authors further show that year-agnostic evaluation, the standard benchmark protocol, overstates performance: for random forest mortality models, the raw-representation AUROC falls from a reported 0.82 on random splits to a true time-averaged 0.76, while the clinical representation is much closer, 0.86 versus 0.85.","pith_inferences":["A testable extension is to construct the Clinical Aggregation mapping using only pre-transition code lists and text descriptions, then measure the 2008 drop; if it exceeds 0.06, the reported robustness depends on knowing the future code system.","The same logic should transfer to other institutions: if a hospital changes EHR vendors, models built on raw vendor-specific codes should be expected to decay unless a semantic mapping exists, which makes standardized clinical vocabularies a deployment requirement.","Because the paper finds smaller subgroups show more erratic drift, fairness evaluations should include subgroup-stratified temporal splits; degradation may not be evenly distributed across protected groups.","The saturation result suggests AUROC on these two tasks is becoming a less sensitive benchmark, so temporal robustness, calibration over time, or harder tasks should be added to standard comparisons."],"forward_implications":["Standard year-agnostic AUROC numbers on MIMIC-III should not be read as expected performance in deployment, especially for raw feature representations.","Deploying models with raw ItemID features across a known EHR system transition will likely incur AUROC drops of roughly 0.29 for mortality and 0.10 for length-of-stay unless the feature representation is made system-independent.","Expert-defined clinical aggregations substantially reduce temporal decay for all tested model classes, so the choice of representation can matter more than the choice of model in non-stationary settings.","Automatic preprocessing such as PCA or UMLS-based concept grouping does not fully close the gap, leaving room for better automatic methods that learn system-spanning semantic mappings.","Both mortality and long length-of-stay tasks appear to saturate in predictive quality given little historical data, suggesting that benchmarking progress on these tasks will require harder targets or additional signals."],"supporting_citations":[{"why":"Contributes the MIMIC-III critical care dataset, including its date-shifting de-identification and the documented 2008 CareVue-to-MetaVision transition that drives the temporal shift.","marker":"Johnson et al. (2016)"},{"why":"Provides the raw feature extraction pipeline and the definitions of the Clinical Aggregation groupings the paper evaluates.","marker":"Wang et al. (2019)"},{"why":"Defines the UMLS-based CUI Code Spanning representation that serves as the automatic feature-grouping competitor.","marker":"Gong et al. (2017)"},{"why":"Supplies the GRU-D model and the simple imputation scheme used for recurrent baselines and missing-data handling.","marker":"Che et al. (2018)"},{"why":"Represents the raw, minimally featurised deep learning approach to ICU prediction that the paper shows is most vulnerable to time shift.","marker":"Purushotham et al. (2018)"},{"why":"Defines the common MIMIC-III mortality and length-of-stay benchmarks whose time-agnostic evaluation protocol the paper critiques.","marker":"Harutyunyan et al. (2017)"}],"fun_headline_variants":["0.29 AUROC drop: EHR models fail on future years","2008 switch causes 0.29 AUROC loss; feature grouping reduces to 0.06","Standard EHR evals overstate models: 0.29 AUROC decay found","Time shift kills EHR model accuracy by 0.29 AUROC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a deployer can produce the expert mapping that links every old measurement code to its new equivalent before the new recording system's codes are observed, since the reported 0.06 and 0.03 AUROC drops assume the mapping spans the 2008 transition.","fun_headline_variants_meta":{"raw":{"variants":["0.29 AUROC drop: EHR models fail on future years","2008 switch causes 0.29 AUROC loss; feature grouping reduces to 0.06","Standard EHR evals overstate models: 0.29 AUROC decay found","Time shift kills EHR model accuracy by 0.29 AUROC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2436,"prompt_tokens":1063,"completion_tokens":1373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":1288}},"tokens_in":679,"tokens_out":1373,"duration_ms":9731,"temperature":1.0,"reasoning_tokens":1288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:38:08.813482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Full History experiment but derive the Clinical Aggregation mapping only from pre-2008 ItemIDs and their text descriptions, then evaluate on 2008-2012 data; if the AUROC drop approaches the raw representation's 0.29 rather than the reported 0.06 for mortality, the mitigation is not achievable prospectively. A second check: repeat the same study on another EHR dataset with a documented system transition and see whether expert-defined clinical concepts keep AUROC declines below raw-code declines.","supporting_citations":[],"review_version":1}