{"id":"bfa143e3-0da9-4077-a558-3c4de01e81cb","arxiv_id":"2412.01973","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A bi-directional LSTM with two parallel phase-processing planes predicts colorectal liver metastases from multi-phase follow-up CTs with AUC 0.79, modestly beating four published baselines.","lead":"Researchers built and tested a deep learning model that reads a patient's past CT scans to predict whether colorectal cancer has spread to the liver, achieving an AUC of 0.79. The model processes arterial and portal-venous contrast phases in parallel, but the dataset is small and not publicly available.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data augmentation duplication before the 10-fold split may leak patient identity across train/test, so the reported 0.790 AUC advantage could be an artifact; a patient-level split check is required.","rationale":"The paper's central claim is an empirical ranking on a private dataset. The most load-bearing condition for that claim is that the evaluation protocol prevents the same patient from contributing to both training and testing. Section 3.1 describes duplicating 60% of positive and 20% of negative cases via SSJ to handle imbalance, and Section 3.2 describes 10-fold cross-validation without specifying patient-level grouping or post-split augmentation. If the duplication is applied before the split, the test fold contains augmented copies of training patients. Because a patient's liver texture is highly consistent across scans, the model can achieve high AUC by recognizing patients rather than by learning generalizable CRLM features. This artifact would not be uniform across models, since MPBD-LSTM has a different temporal structure than the baselines, so the headline '0.790 > 0.765' is not trustworthy until the pipeline is checked. The reader identified irregular scan intervals as the weakest assumption, but that affects interpretation of temporal patterns, not the internal validity of the model comparison. The lack of confidence intervals is also a real limitation, but without the leakage check it is secondary. The paper does have positive elements: code is released, ablations support the design choices, and the medical rationale for using A and V phases is sound. These do not rescue the comparison if the split is contaminated, but they do mean the concern is specifically about the evaluation protocol rather than the method's conceptual basis.","tokens_in":7008,"tokens_out":5646,"duration_ms":54176,"concrete_test":"Re-run the 10-fold cross-validation with patient-level grouped folds: split the 170 patients into 10 folds so no patient appears in more than one fold, apply SSJ duplication and all other augmentations only within each training fold after the split, and recompute the AUCs in Table 2. If MPBD-LSTM's AUC drops below PredRNN-V2's 0.765 or the margin over E3D-LSTM shrinks to less than 0.01, the reported advantage is not robust to leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MPBD-LSTM outperforms all baselines rests on the validity of the 10-fold AUC comparison. Section 3.1 states: 'To handle the imbalanced training dataset, we selected and duplicated 60% of positive cases and 20% of negative cases by applying Standard Scale Jittering (SSJ).' If this duplication is applied to the full 170-patient set before the 10-fold split described in Section 3.2, and the paper gives no indication that it is restricted to training folds, then jittered copies of the same patient can appear in both training and test folds. The classifier could memorize patient-specific liver texture rather than learn generalizable CRLM predictors. Because MPBD-LSTM has a different temporal structure and parameterization than the baselines, leakage would not affect all models equally, so the reported 0.790 versus 0.765 comparison could be an artifact. The paper never states that folds are grouped by patient or that SSJ is applied after splitting. This is the most load-bearing flaw: it invalidates the comparison regardless of model design. Secondary concerns (irregular T0-T2 intervals, missing confidence intervals) are important but do not by themselves invalidate the ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MPBD-LSTM, a multi-plane bidirectional 3D LSTM model for predicting colorectal liver metastases (CRLM) from time-series multi-phase contrast-enhanced CT (CECT) scans. The authors construct a retrospective dataset of 269 patients from two cohorts, select 170 patients with at least three follow-up scans, and use the first three timestamps (T0, T1, T2) of the arterial (A) and portal venous (V) phases as input. They compare MPBD-LSTM against E3D-LSTM, PredRNN-V2, SaConvLSTM, and SimVP under 10-fold cross-validation, reporting an AUC of 0.790 for MPBD-LSTM, the highest among the tested models. Ablation studies examine the contributions of the bi-directional connections, the multi-plane structure, individual timestamps, and individual phases. The code is made publicly available.","tokens_in":7256,"tokens_out":2692,"duration_ms":25692,"significance":"If the reported results hold, the paper makes a useful contribution to medical image analysis by demonstrating a concrete architecture for handling 5D spatiotemporal data (time, phase, and 3D volume) in a clinically relevant prediction task. The construction of a private CECT dataset, the public release of code, and the ablation studies on phase and timestamp usage are valuable assets. The central claim is an empirical benchmark result rather than a theoretical derivation, and the comparison across four strong baselines is informative for practitioners. However, the significance is currently limited by the lack of statistical rigor in the evaluation and by unresolved questions about data preprocessing leakage, which could materially affect the reported ranking.","major_comments":[{"comment":"The paper does not specify whether the data augmentation duplication is applied before or after the 10-fold split described in Section 3.2. Section 3.1 states that 'we selected and duplicated 60% of positive cases and 20% of negative cases by applying Standard Scale Jittering (SSJ)' on the full set of 170 patients. If this duplication occurs before splitting, jittered copies of the same patient can appear in both training and test folds, allowing the model to memorize patient-specific liver texture rather than learn generalizable CRLM predictors. Because MPBD-LSTM and the baselines have different parameterizations, such leakage would not affect all models equally, so the reported AUC advantage of 0.790 over 0.765 could be an artifact. The authors must clarify that augmentation is performed only on each training fold after splitting, or alternatively report results with patient-level grouped folds and post-split augmentation.","section":"Section 3.1"},{"comment":"The central comparison reports only point AUC values without confidence intervals, standard deviations across folds, or statistical significance tests. The gap between MPBD-LSTM (0.790) and PredRNN-V2 (0.765) is modest, and on a dataset of only 170 patients this difference may fall within the noise of 10-fold cross-validation. The authors should report per-fold AUC values, confidence intervals (e.g., bootstrap or DeLong), and significance tests (e.g., paired DeLong test or Wilcoxon signed-rank test) for the main comparison and for the ablations in Tables 3 and 4.","section":"Section 4, Table 2"},{"comment":"The input construction treats T0, T1, and T2 as aligned timestamps across patients, yet Section 2.1 states that patients have between 2 and 6 follow-up scans at variable intervals after surgery. If the temporal spacing between the first three scans differs across patients, the model may learn patterns related to follow-up scheduling rather than disease progression. The authors should report the distribution of inter-scan intervals and either align scans by actual time or perform a sensitivity analysis restricted to patients with similar follow-up schedules.","section":"Section 2.1 and Section 3.1"},{"comment":"Several key design choices appear to be tuned on the same 170-patient dataset without a nested validation procedure: the number of timestamps (three), the augmentation duplication ratios (60% positive, 20% negative), the input slice depth (64), and the LSTM hidden unit count (16). This creates an overfitting risk that is not addressed by the reported 10-fold cross-validation. The authors should either use a nested cross-validation or an independent validation set for these choices, or explicitly justify each choice by reference to prior work rather than dataset-specific tuning.","section":"Section 3.1 and Section 3.2"}],"minor_comments":[{"comment":"The word 'bi-directioal' appears to be a typo for 'bi-directional'.","section":"Abstract"},{"comment":"The table caption contains a typo: 'Characreristics' should be 'Characteristics'.","section":"Table 1"},{"comment":"Equation (1) uses a general timestamp 't' while the surrounding text and Eq. (2) use 't0'; please make the notation consistent and clarify that Eq. (1) holds for every timestamp.","section":"Section 2.2, Eq. (1)-(2)"},{"comment":"The description of the duplication process is ambiguous: it is not clear whether 'duplicated' means each selected case is copied once or multiple times, nor how the final class counts after augmentation compare to the original 49 positive and 121 negative cases. Please state the exact post-augmentation counts.","section":"Section 3.1"},{"comment":"The text refers to supplementary material for accuracy, sensitivity, and specificity, but no supplementary material is available in the arXiv submission; please include these metrics in the main paper or make the supplement publicly available.","section":"Section 4, Table 2"},{"comment":"The claim that 'small liver size is present in most of the false negative cases' is not supported by quantitative data; please provide a statistical comparison of liver sizes between correctly and incorrectly predicted positive cases.","section":"Section 4, Error Analysis"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the potential data leakage from augmentation before the cross-validation split, which, if confirmed, would invalidate the reported AUC ranking. However, this is fixable by rerunning the experiments with a patient-level split and augmentation applied after splitting; therefore major revision rather than rejection is appropriate. The paper would also benefit from a statistical treatment of the AUC comparisons, as the current point estimates are not convincing on a dataset of this size."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid empirical study with a genuinely new longitudinal multi-phase CECT dataset (269 patients across two cohorts, 170 used in the experiments) and a sensible architecture tweak: a bidirectional LSTM on two parallel phase planes, built from E3D-LSTM. The ablation study is clean and supports the design choices: removing the multi-plane structure drops AUC from 0.790 to 0.774, removing the bidirectional connection drops it to 0.768, and temporal/phase ablations behave as you would expect. The error analysis is thoughtful. The code is on GitHub. That is real, credit-worthy work.\n\nThe soft spot is the preprocessing order. Section 3.1 says they duplicated 60% of positive cases and 20% of negative cases with Standard Scale Jittering to handle imbalance. Section 3.2 then describes 10-fold cross-validation, but never states that the duplication is restricted to training folds or that folds are grouped by patient. If jittered copies of the same patient end up in both train and test, the classifier can memorize patient-specific liver texture rather than learn generalizable predictors. Since MPBD-LSTM has a different temporal structure than the baselines, leakage would not affect all models equally, so the reported 0.790 versus 0.765 advantage could be an artifact. This is the load-bearing issue, and it is entirely fixable: report patient-level splits, apply augmentation only after splitting, or show per-fold results with identical folds.\n\nSecondary concerns are real but smaller. There are no confidence intervals or significance tests; the AUC gap over PredRNN-V2 is 0.025 and could be within noise. The time axis treats T0, T1, T2 as aligned across patients even though scan intervals vary, so temporal patterns may reflect scheduling rather than disease progression. The dataset is \"available upon request\" with no public release plan, so independent verification is not currently possible.\n\nOverall, this is a clinically relevant task, the dataset is new, and the architecture integration appears sound. The missing split details are a methodological ambiguity rather than evidence of dishonesty, but they decide whether the central claim holds. I would send this to peer review, because the issues are addressable and the contribution deserves referee time. If the split ambiguity is resolved and external validation appears, it becomes a reasonable addition to the CRLM prediction literature.","headline":"A plausible new CRLM prediction model and dataset, but the missing split details could sink the central comparison; worth a careful referee.","tokens_in":7784,"tokens_out":1973,"would_cite":false,"duration_ms":19226,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a second processing plane and bidirectional time flow to a 3D LSTM improves early prediction of colorectal liver metastases from serial contrast-enhanced CT.","keywords":["colorectal liver metastasis","contrast-enhanced CT","bidirectional LSTM","multi-plane architecture","time-series prediction","medical image classification","5D CT data","MPBD-LSTM"],"falsifier":"A reader could re-run the same 10-fold protocol on a cohort where every patient has scans at exactly the same post-surgery intervals; if the AUC advantage of MPBD-LSTM over PredRNN-V2 shrinks or vanishes, the claim that the bidirectional multi-plane structure is responsible would be falsified.","tokens_in":6816,"feed_emoji":"🩻","tokens_out":7349,"duration_ms":61798,"temperature":0.7,"pith_summary":"Colorectal cancer patients are followed with repeated contrast-enhanced CT scans after surgery, and the paper asks whether a deep learning model can read that scan history to flag patients who will develop liver metastases within two years. The authors assembled a two-cohort dataset of 269 patients (170 with at least three scans) with arterial and portal venous phase volumes at three follow-up times, and compared several video-prediction networks adapted to this five-dimensional data. Their central finding is that treating the two contrast phases as parallel processing planes and letting a 3D LSTM read each plane both forward and backward in time gives the best prediction, AUC 0.790, ahead of PredRNN-V2 (0.765), E3D-LSTM (0.755), SaConvLSTM (0.721), and SimVP (0.662). This matters because earlier detection of liver metastasis is a route to earlier treatment, and because the result identifies two design choices, phase separation and bidirectional temporal reading, that can be tested in other multi-phase imaging problems.","feed_headline":"Two-plane bidirectional LSTM hits top AUC 0.790 for liver metastasis","feed_subtitle":"Serial CT of 170 patients: parallel phase planes plus bidirectional LSTM beats standard video models.","key_machinery":"The load-bearing object is MPBD-LSTM, built from 3D-LSTM modules. Each module contains two E3D-LSTM cells, one forward and one backward, whose hidden states are combined by summation, $y_{v,t} = \\overrightarrow{h}_{v,t} \\oplus \\overleftarrow{h}_{v,t}$. Two such planes process the arterial (A) and portal venous (V) phases separately with the same backbone and hyperparameters, and the hidden states from the two planes are averaged after the LSTM layers before classification. This construction isolates the two hypotheses the paper tests: phase should be a parallel structure rather than a concatenation channel, and temporal dynamics should be read in both directions. The ablation study then removes each component in turn to attribute the AUC gain, including a variant with early inter-plane connections that performs slightly worse and is therefore discarded.","core_discovery":"The central claim is an architectural result: the best way to extend a 4D spatiotemporal model to 5D multi-phase contrast-enhanced CT is not to concatenate the arterial and portal venous phases into channels, but to run one E3D-LSTM backbone per phase and fuse their outputs after the recurrent layers, while replacing the unidirectional temporal pass with a forward-and-backward pair whose hidden states are summed. On the paper's 170-patient evaluation, MPBD-LSTM reaches AUC 0.790, and the ablations attribute the gain to both choices: removing the multi-plane structure and concatenating phases lowers the score to 0.774, and switching the bidirectional connections back to unidirectional lowers it to 0.768. The time-series and multi-phase ablations further show that using all three timestamps beats any single timestamp, and that the venous phase alone (0.752) is much stronger than the arterial phase alone (0.653), so the full five-dimensional input is what the model needs.","pith_inferences":["A testable extension the authors did not run: re-index the time axis by days since surgery rather than visit number. If the AUC drops, the model is partly learning follow-up scheduling; if it holds, the learned features are tied to biological change.","The phase-plane design should transfer to other multi-phase or multi-sequence imaging, such as multiparametric MRI, where the same choice, parallel planes versus channel concatenation, can be evaluated directly.","The false-negative concentration in small livers suggests a preprocessing remedy: liver-volume normalization, or training with size-stratified losses, might recover a meaningful share of the missed positives."],"forward_implications":["On the reported data, MPBD-LSTM outperforms all four baselines by AUC, with the largest gap over the CNN-only SimVP (0.790 vs 0.662), suggesting recurrent temporal models fit this task better than pure convolutional translators.","The ablation results imply that concatenating arterial and venous phases loses slice-position correspondence; using two parallel planes recovers about 0.016 AUC over concatenation.","The ablation results imply that bidirectional temporal connections add about 0.022 AUC over the unidirectional version, supporting the value of reading the scan history in both time directions.","Single-timestamp performance rises from T0 (0.660) to T2 (0.709), and using all three timestamps (0.790) beats any single time, supporting the clinical value of serial follow-up scans.","The portal venous phase alone (0.752) dominates the arterial phase alone (0.653), consistent with the radiological view that venous-phase blood supply is more informative for colorectal liver metastasis."],"supporting_citations":[{"why":"Supplies the E3D-LSTM cell architecture that MPBD-LSTM doubles into forward and backward pairs.","marker":"[12]"},{"why":"Provides the medical-imaging precedent for bidirectional recurrent connections that MPBD-LSTM adapts.","marker":"[2]"},{"why":"Defines the summation combination of forward and backward hidden states used in each 3D-LSTM module.","marker":"[3]"},{"why":"PredRNN-V2 is the strongest baseline at AUC 0.765 and is compared against MPBD-LSTM.","marker":"[14]"},{"why":"SimVP is the pure-CNN baseline whose lower AUC supports the paper's recurrent-model conclusion.","marker":"[4]"},{"why":"SaConvLSTM is a self-attention ConvLSTM baseline in the comparison.","marker":"[9]"},{"why":"Provides the medical basis that the venous phase is more informative for colorectal liver metastasis.","marker":"[15]"},{"why":"Mixup is one of the data augmentation techniques used in training.","marker":"[17]"},{"why":"Standard Scale Jittering is used to rebalance the positive and negative training cases.","marker":"[5]"},{"why":"Spline Interpolated Zoom uniformly selects the 64 CT slices used as model input.","marker":"[18]"}],"fun_headline_variants":["Bidirectional LSTM fusion beats channel-concat for liver mets","MPBD-LSTM: 5D CT model reaches AUC 0.79 for liver metastases","Per-phase LSTM fusion outperforms channel-wise 5D CT for liver mets","MPBD-LSTM: early liver metastasis detection at AUC 0.79","LSTM on multi-phase CT predicts liver mets with 0.79 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model's evaluation assumes the first three follow-up scans T0, T1, T2 can be treated as the same aligned time series for every patient, even though the dataset allows 2 to 6 scans at irregular post-surgery intervals (Section 2.1, Section 3.1); if scan timing correlates with disease or hospital routine, the temporal signal may be partly scheduling, not biology.","fun_headline_variants_meta":{"raw":{"variants":["Bidirectional LSTM fusion beats channel-concat for liver mets","MPBD-LSTM: 5D CT model reaches AUC 0.79 for liver metastases","Per-phase LSTM fusion outperforms channel-wise 5D CT for liver mets","MPBD-LSTM: early liver metastasis detection at AUC 0.79","LSTM on multi-phase CT predicts liver mets with 0.79 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002187,"raw_usage":{"total_tokens":8491,"prompt_tokens":983,"completion_tokens":7508,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":7402}},"tokens_in":599,"tokens_out":7508,"duration_ms":259103,"temperature":1.0,"reasoning_tokens":7402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:57:24.881861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could re-run the same 10-fold protocol on a cohort where every patient has scans at exactly the same post-surgery intervals; if the AUC advantage of MPBD-LSTM over PredRNN-V2 shrinks or vanishes, the claim that the bidirectional multi-plane structure is responsible would be falsified.","supporting_citations":[{"cited_title":"In: International conference on learning representations (2019)","cited_arxiv_id":null,"evidence_quote":"Supplies the E3D-LSTM cell architecture that MPBD-LSTM doubles into forward and backward pairs."},{"cited_title":"Ad- vances in neural information processing systems 29 (2016)","cited_arxiv_id":null,"evidence_quote":"Provides the medical-imaging precedent for bidirectional recurrent connections that MPBD-LSTM adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PredRNN-V2 is the strongest baseline at AUC 0.765 and is compared against MPBD-LSTM."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"SimVP is the pure-CNN baseline whose lower AUC supports the paper's recurrent-model conclusion."},{"cited_title":"In: Proceedings of the AAAI conference on artificial intel- ligence","cited_arxiv_id":null,"evidence_quote":"SaConvLSTM is a self-attention ConvLSTM baseline in the comparison."},{"cited_title":"World journal of gastroenterology: WJG 17(42), 4654 (2011)","cited_arxiv_id":null,"evidence_quote":"Provides the medical basis that the venous phase is more informative for colorectal liver metastasis."},{"cited_title":"In: Predictive Intelligence in Medicine: Third International Workshop, PRIME 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 8, 2020, Proceedings 3","cited_arxiv_id":null,"evidence_quote":"Spline Interpolated Zoom uniformly selects the 64 CT slices used as model input."}],"review_version":1}