{"id":"29d044a6-50b3-4e63-8cfb-4e65a4dc51d1","arxiv_id":"2412.00105","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fusion of LSTM and TCN models built from sampling-frequency feature subsets achieved only modest extubation-failure prediction (AUC-ROC about 0.6) on 4,701 MIMIC-IV patients.","lead":"This study develops LSTM, TCN, and LightGBM models to predict whether ICU patients fail extubation, using six hours of MIMIC-IV data per patient. All models performed only modestly, with AUC-ROC around 0.6, but the paper documents how synthetic data from irregular monitoring can bias temporal models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Section 3.1 outcome definition counts any O2 flow or NIV within 6h post-extubation as failure, inflating the rate to 32.8% versus the cited 10-20%; if these are routine supportive care, the labels, and thus the bias-mitigation claim, rest on mislabeled ground truth.","rationale":"The reader's weakest assumption is also the most load-bearing concern in this work. The outcome definition enters at the very start of the modeling pipeline (Section 3.1) and determines every label. If 'receiving O2 flow' within 6 hours after extubation is counted as failure, then the observed 32.8% failure rate is far higher than the 10-20% reported in the broader literature cited by the paper itself. Routine post-extubation oxygen is standard in many ICUs, so this criterion likely reclassifies many successful extubations as failures. Under that scenario, the central descriptive claim—that frequency-stratified subsets and a fused LSTM/TCN eliminate complete class bias—is a statement about a mislabeled dataset: the models may be learning to predict a composite outcome that is not extubation failure. The concrete test of re-annotating with only reintubation or death within 48 hours would directly settle whether the claim survives a standard definition. I considered the alternative concern that the preprocessing strategy was revised on the same fixed test set; this is real and important, but it affects the magnitude and generalizability of the performance estimate, whereas the label definition affects the validity of the outcome itself. The paper is honest and detailed, with limitations clearly stated, and the weak final performance is not hidden. The conditional verdict remains appropriate: the work is a useful methodological caution, but its central claim is not established under a conventional extubation failure definition. No verdict change is needed.","tokens_in":44179,"tokens_out":5756,"duration_ms":55456,"concrete_test":"Re-annotate the cohort using only the two standard criteria (reintubation or death within 48 hours), dropping the ventilatory-support-within-6h criterion. Keep the same feature preprocessing, train/test split, and the fused LSTM/TCN/LightGBM pipeline, then report the resulting failure rate, AUC-ROC, F1, and whether vanilla LSTM/TCN still show complete class bias. If the failure rate drops to roughly 10-20% and the fused models still remove bias with AUC-ROC near 0.6, the outcome definition is not the decisive weak point. If the failure rate becomes substantially imbalanced or the fused models' discrimination changes materially, the central claim is conditional on a nonstandard label and must be reframed or re-evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines extubation failure as reintubation or death within 48 hours, or any ventilatory support (NIV, O2 flow, CPAP, BiPAP) within 6 hours after extubation. The inclusion of 'O2 flow' is the most problematic component: supplemental oxygen is routinely administered after extubation as standard supportive care, not necessarily as a marker of failed liberation. This criterion produces a 32.0% failure rate before data exclusions and 32.8% in the final cohort, roughly double the 10-20% the paper itself cites in Section 2.2. If routine oxygen support is not true extubation failure, then a large fraction of positive labels are incorrect. The central claim—that frequency-stratified feature subsets and a fused LSTM/TCN remove the complete class bias seen in vanilla temporal models—is then a statement about models trained on partially mislabeled data. The observed transition from complete bias (AUC-ROC ~0.5) to modest discrimination (AUC-ROC ~0.6) could be an artifact of this nonstandard label rather than a robust property of the preprocessing strategy. The paper acknowledges the lack of consensus on outcome definitions and defends its choice via clinician input, but the risk that routine O2 flow is being counted as failure is not addressed with any sensitivity analysis. This is a correctness risk about the validity of the ground truth, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops and evaluates an end-to-end machine learning pipeline for predicting extubation failure from 6 hours of pre-extubation data in MIMIC-IV (4,701 patients). Three feature sets, constructed by cross-referencing literature popularity with clinical availability in the WAVE study, are used to train LSTM, TCN, and LightGBM models. Initial temporal models trained on uniformly resampled data are reported to be completely biased toward one class; the authors then group features by sampling frequency, resample each subset at a bespoke rate, mask missing values, and combine three subnetworks in a fused decision architecture. The final models attain AUC-ROC around 0.6 and F1 below 0.5, with no architecture, feature set, or static-data configuration clearly superior. The paper honestly reports weak discrimination and frames its contribution as a set of preprocessing strategies that remove complete class bias and as a foundation for future work.","tokens_in":44417,"tokens_out":3798,"duration_ms":37109,"significance":"If the empirical claims hold, the paper makes a useful cautionary contribution: it documents how naive uniform resampling of sparsely and irregularly sampled ICU time series can induce complete class bias, and it demonstrates a frequency-stratified, masking-based alternative that restores some discriminatory signal. The study also releases code on GitHub, involves clinician input in cohort and feature decisions, and reports negative results without overclaiming clinical utility. However, the significance is tempered because the headline result is modest (AUC-ROC about 0.6), the evaluation is based on point estimates from a single split, and the outcome-label definition is nonstandard and potentially mislabels routine post-extubation oxygen support as failure. The paper's value is therefore primarily methodological and cautionary, rather than a demonstrated clinically actionable predictor.","major_comments":[{"comment":"The outcome definition is load-bearing and needs a sensitivity analysis. Extubation failure is defined as reintubation or death within 48 hours, or any ventilatory support including 'O2 flow' within 6 hours post-extubation. This yields a 32.8% failure rate in the final cohort, roughly double the 10–20% range the paper itself cites in §2.2. Supplemental oxygen is routinely administered after extubation as supportive care, so a large fraction of positive labels may not represent true failure. Since the central claim is that the frequency-stratified preprocessing removed complete class bias, that claim is currently about models trained and evaluated on potentially mislabeled ground truth. The authors acknowledge the lack of consensus but provide no sensitivity analysis. I request ablations of the label: (a) reintubation/mortality only, (b) exclude the O2-flow criterion, and (c) exclude all ventilatory-support-only positives. If the bias-removal result is robust to these label variations, the claim is substantially strengthened; if not, the conclusion needs to be restricted to the authors' specific label definition.","section":"§3.1, Data annotation"},{"comment":"All reported results are point estimates from a single 80/20 stratified split, with no confidence intervals, bootstrap replicates, or repeated-seed experiments. Differences such as Fused LSTM AUC-ROC 0.6567 versus Fused TCN 0.6116 on Feature Set 1, or the modest AUC-ROC changes when static data is added, may be within sampling noise. Given the paper's central comparison is between preprocessing strategies and architectures, the absence of uncertainty quantification makes it impossible to know whether the observed differences are real. Please add bootstrap confidence intervals for the test-set metrics, or repeated cross-validation with multiple seeds, and state whether architecture and static-data comparisons are statistically distinguishable.","section":"§5, Table 5.1 and §3.2, train/test split"},{"comment":"The claim that frequency-stratified subsetting 'removed complete class bias' is confounded by several simultaneous changes between the vanilla and fused pipelines. In moving from the initial LSTM/TCN setup to the fused setup, the authors also change the resampling interval per subset, introduce NaN masking, replace the single network with three parallel subnetworks, remove Ventilator Mode, and drop the SpO2:FiO2 and PaO2:FiO2 engineered ratios. Any one of these changes could be responsible for the observed improvement. To support the attribution, I request an ablation that varies one factor at a time: for example, a vanilla LSTM/TCN trained on uniformly resampled data but with the same masking, or a fused architecture trained on the original 30-minute uniform resampling. Without such experiments, the paper's headline causal claim is not established.","section":"§4.2, 'Devising a new approach' and §5, Table 5.1"},{"comment":"The paper does not report the actual proportion of synthetic (interpolated) values within each frequency subset. For the low-frequency subset, features are sampled on average about 0.5 times per 6-hour window and are resampled to a 2-hour interval (sequence length 4); with only one real observation, most time steps must be filled by the start/end imputation logic and linear interpolation. The authors stratify the train/test split by an overall synthetic-data proportion but never state the per-subset proportions that motivated the whole approach. Please quantify synthetic fractions per subset before and after the new strategy, and show that the 'low' subset is not still dominated by synthetic patterns. This is needed to substantiate the claim that the strategy actually minimised synthetic data impact rather than merely redistributing it.","section":"§4.2, Low/Medium/High frequency subsets and §3.2, synthetic data stratification"}],"minor_comments":[{"comment":"The document is formally a thesis ('A Thesis submitted in fulfillment...'), and the abstract contains the phrase 'This thesis highlights the challenges...'. For arXiv publication in cs.LG, please reframe as a research paper and remove thesis-specific formatting artifacts.","section":"Title page / Abstract"},{"comment":"There are numerous typographical errors and formatting glitches, including 'Extubation F ailure' in the title, 'Confustion matrix' in §4.1, 'F eatures' in section headings, and several run-together words such as 'W edecided'. A careful proofread is needed.","section":"Throughout"},{"comment":"The cohort flow is confusing: 5,970 patients before extraction become 4,701 after extraction, but the success/failure counts of 4,059 and 1,911 correspond to 5,970. Please present a clear CONSORT-style flow diagram with numbers at each exclusion stage, including the 1,269 patients with no recorded data.","section":"§3.1, Data extraction"},{"comment":"The text in §4.1 says features such as O2 saturation are recorded 'approximately once an hour,' and Table 3.2 confirms an average frequency of 6.6 per 6-hour window, which is consistent. However, the low-frequency features PH, PaCO2, and PaO2 are described as 'once every 12 hours' while the table lists about 0.5 per window; please harmonise the wording with the table.","section":"Table 3.2 and §2.2"},{"comment":"A GitHub link is provided, but the paper does not describe the repository structure, software versions, or how to reproduce the experiments. Given the emphasis on rigor and a conditional acceptance, please add a reproducibility statement with environment details and a brief description of the release.","section":"§4.3, Implementation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an MSc thesis submitted as an arXiv paper; the editor may wish to consider whether the thesis format, with its extended narrative of failed initial experiments, is appropriate for a journal-style publication or should be condensed. The label-definition concern in §3.1 is the most serious issue and should be handled with explicit sensitivity analyses rather than a statement about clinician consensus. I do not see grounds for rejection: the paper is transparent about weak performance, releases code, and documents a reproducible-looking bias phenomenon. The revision burden is substantial but within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an honest, iterative MSc thesis that reports weak results (AUC ~0.6, F1 <0.5) but makes a useful empirical point: resampling irregular ICU vitals to a common 30-minute grid produces so much synthetic data that LSTM and TCN models collapse to predicting a single class, and grouping features by their native sampling frequency and resampling each group separately removes that complete bias. That narrative is supported by their own experiments and is worth knowing for anyone building temporal models on MIMIC.\n\nWhat's new: TCN for extubation failure specifically, a fused LSTM/TCN with frequency-stratified feature subsets, masking for missing values, and a train/test split stratified by synthetic data proportion. The paper also does a thorough literature review and is unusually transparent about the iterative failure process. Credit where due: they report the negative results plainly, provide a GitHub link, and do not overstate the final models as clinically usable. The claim is descriptive, not a claim of utility.\n\nThe biggest soft spot is the outcome definition in Section 3.1. They count any O2 flow within 6h post-extubation as failure, which inflates the failure rate to 32.8% versus the 10-20% they cite; routine post-extubation oxygen is standard supportive care, not necessarily failure. If a large fraction of positive labels are wrong, then the observed improvement—from complete bias to AUC 0.6—is a statement about models trained on questionable ground truth. No sensitivity analysis addresses this. Also, results are point estimates from one split with no confidence intervals, and the preprocessing strategy was revised on the same fixed test set, which risks selection effects. The title claims \"actionable,\" which is not supported by F1 <0.5.\n\nWho it's for: readers working on MIMIC temporal models, especially on label definitions and resampling artifacts. It deserves a serious referee, but conditional on the author addressing the label definition with sensitivity analyses (e.g., excluding O2-only cases) and adding error bars. Recommend engaging with it.","headline":"Honest empirical study showing that naive resampling of irregular ICU data creates complete class bias in temporal models; useful as a methodological caution, but the outcome definition counting routine O2 as failure undermines the ground truth.","tokens_in":45009,"tokens_out":2022,"would_cite":false,"duration_ms":19522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T05","92C50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Uniform resampling of irregular ICU data makes LSTM and TCN models collapse to single-class predictions; stratifying features by sampling frequency and fusing their outputs restores weak discrimination (AUC-ROC around 0.6).","keywords":["extubation failure","mechanical ventilation","MIMIC-IV","temporal deep learning","LSTM","temporal convolutional network","synthetic data","clinical interpretability"],"falsifier":"Re-label the same MIMIC-IV cohort with failure defined only as reintubation or death within 48 hours, dropping the 6-hour ventilatory-support criterion, and retrain the fused LSTM/TCN pipeline; if AUC-ROC jumps substantially above 0.6, the broad definition's mislabelled failures were the main performance ceiling, while if AUC-ROC stays near 0.6 the weak discrimination is intrinsic to the data.","tokens_in":43901,"feed_emoji":"🫁","tokens_out":5764,"duration_ms":50197,"temperature":0.7,"pith_summary":"This thesis argues that the main obstacle to predicting extubation failure from ICU time series is not the choice of neural architecture but the way irregularly recorded data is resampled into synthetic sequences. The author shows that when all features are uniformly interpolated to 30-minute steps, both LSTM and TCN models collapse into predicting every patient as the same class, regardless of hyperparameter tuning or added static features. Splitting features into low-, medium-, and high-frequency subsets, resampling each at a rate that respects its observed density, masking absent values, and fusing the three sub-models removes that total bias. On 4,701 MIMIC-IV patients the resulting models still discriminate only weakly, with AUC-ROC around 0.6 and F1 below 0.5. The point matters because extubation decisions are high-stakes, and the finding redirects attention from architecture toward data-synthesis and outcome labelling.","feed_headline":"Synthetic data, not model choice, broke ICU extubation predictors","feed_subtitle":"Stratifying features by sampling rate and fusing LSTM/TCN removed total bias, yet AUC-ROC stayed near 0.6.","key_machinery":"The central mechanism is the frequency-stratified feature subset with a fused decision head. Dynamic features are divided into groups based on average observation frequency in the 6-hour window, resampled at rates matched to their observed density, and passed through separate LSTM or TCN branches; static features are processed by a feed-forward network; the last valid output of each branch is concatenated and fused into a sigmoid prediction. Masking tells the models to ignore time steps where no real observation existed, avoiding wholesale synthetic imputation. This machinery is what converts complete single-class bias into weak but genuine discrimination.","core_discovery":"The paper's central claim is that synthetic data handling, not model architecture, was the dominant driver of the observed failures. Vanilla LSTM and TCN models trained on data uniformly resampled to 30-minute intervals showed complete bias, predicting all patients into one class; this persisted through grid search, Bayesian optimisation, static data inclusion, and architecture changes. The author then grouped dynamic features into low, medium, and high sampling-frequency subsets, resampled each subset at a bespoke rate (2 hours, 1 hour, and 30 minutes respectively), masked missing values instead of imputing them, and processed each subset through a separate LSTM or TCN whose outputs were fused. This frequency-stratified, fused design eliminated the total class bias, but final discrimination remained modest across all architectures and feature sets, with AUC-ROC around 0.6 and F1 consistently below 0.5.","pith_inferences":["The frequency-stratified, mask-and-fuse recipe could be tested as a general preprocessing baseline for any irregularly sampled electronic health record time series before attributing poor performance to the model family.","A cleaner outcome definition (reintubation or death within 48 hours only) might raise apparent discrimination; if AUC-ROC rises markedly, much of the observed ceiling is label noise rather than missing predictive signal.","Because the fused design keeps frequency subsets separate until the final layer, cross-frequency interactions (for example, a low-frequency blood gas paired with a high-frequency oxygen saturation) are only available very late; an input-level multi-resolution fusion would directly test whether those interactions carry signal.","The near-zero ablation deltas suggest the models are insensitive to single features; permutation-based importance across entire subsets would test whether the signal lives in groups of features rather than in individuals."],"forward_implications":["If the paper is right, future LSTM/TCN studies on irregular ICU data should stratify features by observation frequency before declaring an architecture inadequate; uniform resampling can by itself produce total class bias.","The fused decision system removes complete bias but caps discrimination at AUC-ROC around 0.6, so the temporal architectures as configured here are not yet sufficient for clinical use in extubation failure prediction.","Static data and additional features do not reliably move AUC-ROC; parsimonious feature sets perform comparably to larger ones, echoing earlier findings that smaller feature sets can match larger ones.","Feature ablation shows near-zero change for most individual features, meaning single-feature interpretability offers little actionable guidance on this dataset.","Because the study defines failure to include any ventilatory support within 6 hours, the cohort failure rate is 32.8%, far above the commonly cited 10–20%, so comparisons with prior extubation models must account for label definition."],"supporting_citations":[{"why":"Supplies the MIMIC-IV electronic health records from which the 4,701-patient cohort is extracted.","marker":"[94-96]"},{"why":"Closest prior art: the only peer-reviewed temporal-model study of extubation failure, used as the comparison point and as a caution about imputation and windowing choices.","marker":"[80]"},{"why":"Provides the fused TCN/LSTM-with-FFNN architecture and missingness-masking idea that the thesis adapts to extubation failure.","marker":"[77]"},{"why":"Provides the generic TCN block design with causal dilated convolutions and residual connections used for the TCN models.","marker":"[115]"},{"why":"Demonstrates a MIMIC-based extubation model with SHAP-based feature selection and deployment, serving as the main non-temporal benchmark for feature-set comparisons.","marker":"[49]"},{"why":"Defines the LightGBM baseline used to compare temporal versus non-temporal models.","marker":"[53]"},{"why":"Contributes the WAVE study feature set and prior ventilation-weaning work used as the clinical benchmark in feature selection.","marker":"[97]"},{"why":"Supplies the Beacon Care/WAVE clinical feature set that anchors the cross-referencing feature selection strategy.","marker":"[98]"}],"fun_headline_variants":["Synthetic data is the real culprit in ICU extubation AI failure","Fused LSTM/TCN kills class bias, but AUC stays 0.6","Bias-free extubation model still capped at AUC 0.6","Data handling, not AI, keeps extubation AUC at 0.6","Frequency-stratified fusion fixes bias, not performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The outcome definition is load-bearing: extubation failure is labelled as reintubation or death within 48 hours, or any ventilatory support (NIV, oxygen flow, CPAP, BiPAP) within 6 hours after extubation, which yields a 32.8% failure rate, and if routine post-extubation support is not true failure then a large fraction of labels are wrong.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data is the real culprit in ICU extubation AI failure","Fused LSTM/TCN kills class bias, but AUC stays 0.6","Bias-free extubation model still capped at AUC 0.6","Data handling, not AI, keeps extubation AUC at 0.6","Frequency-stratified fusion fixes bias, not performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001398,"raw_usage":{"total_tokens":5705,"prompt_tokens":1046,"completion_tokens":4659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":4563}},"tokens_in":662,"tokens_out":4659,"duration_ms":30100,"temperature":1.0,"reasoning_tokens":4563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:52:38.563469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the same MIMIC-IV cohort with failure defined only as reintubation or death within 48 hours, dropping the 6-hour ventilatory-support criterion, and retrain the fused LSTM/TCN pipeline; if AUC-ROC jumps substantially above 0.6, the broad definition's mislabelled failures were the main performance ceiling, while if AUC-ROC stays near 0.6 the weak discrimination is intrinsic to the data.","supporting_citations":[{"cited_title":"Decision support system to evaluate ventilation in the acute respiratory distress syndrome (DeVENT study)—trial protocol","cited_arxiv_id":null,"evidence_quote":"Supplies the Beacon Care/WAVE clinical feature set that anchors the cross-referencing feature selection strategy."}],"review_version":1}