{"id":"59a939ed-cbb0-4a5b-9ffa-063f5ff053e1","arxiv_id":"2411.14407","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining angiographic imaging, patient biomarkers, and aneurysm morphology with SMOTE/SMOTENC augmentation improved intermediate fusion AUROC to 0.85 in a 340-patient study.","lead":"This paper tests whether combining imaging data, patient history, and aneurysm shape improves predictions of aneurysm treatment success. It finds that adding synthetic examples to balance the classes raises a fusion model's AUROC from 0.68 to 0.85.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AUROC gain to 0.85 depends on an unspecified augmentation/cross-validation protocol; if SMOTE/SMOTENC is applied before splitting, leakage could invalidate the central claim.","rationale":"I read the paper in good faith: it reports a concrete empirical comparison of fusion architectures and augmentation methods on a clinical dataset, and the central result is the intermediate-fusion SMOTE/SMOTENC AUROC of 0.85. The entire conclusion that augmentation substantially enhances performance rests on that number. For that number to be credible, the evaluation must prevent synthetic training examples from leaking into validation, and the manuscript never specifies when augmentation is applied. This is not a disagreement with a consensus modeling choice; it is an internally unresolved methodological precondition. The reader's weakest assumption identified exactly this issue, and I agree with that assessment. I would not change the CONDITIONAL verdict: the paper should be accepted only if the authors supply a leakage-free rerun or otherwise demonstrate that augmentation was applied within folds. Without that, the 0.85 result could simply reflect memorization of augmented training data. No ad hominem is intended; the issue is an underspecified protocol, not suspected misconduct.","tokens_in":741,"tokens_out":1142,"duration_ms":64245,"concrete_test":"Ask the authors to rerun the intermediate-fusion experiment with a strict leakage-free protocol: for each of the 20 Monte Carlo splits, fit the standard scaler, one-hot encoder, and all augmentation (SMOTE/SMOTENC) using only the training fold, then evaluate on the untouched validation fold. If the resulting AUROC remains at or above 0.80, the leakage concern is refuted; if it falls to the 0.68–0.75 range, the reported 0.85 gain is largely an artifact of evaluation leakage. The same check should be reported alongside the class-imbalance ratio (minority n) for interpretability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SMOTE/SMOTENC raises intermediate-fusion AUROC from 0.68±0.1 to 0.85±0.06. This is meaningful only if synthetic minority samples are generated exclusively from training folds and never enter validation or test folds. Section 2.3 describes SMOTE/SMOTENC, RandomOverSampler, and ADASYN but does not state when augmentation is applied relative to the 20-split Monte Carlo cross-validation mentioned in the abstract. Figure 1 shows augmentation after preprocessing and before model training, which is consistent with either per-fold application or application to the entire dataset before splitting. If the latter, validation AUROC can be inflated: RandomOverSampler duplicates real minority instances, so the same patient may appear in both training and validation folds; SMOTE/SMOTENC interpolates from nearest neighbors, and if those neighbors are drawn from the full dataset, synthetic examples encode validation information. Because the cohort has only 340 patients and the class-imbalance ratio is not reported, the minority class is likely small, making this leakage potentially decisive. The paper therefore does not currently rule out the most natural alternative explanation for the 0.85 result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates a multimodal machine learning framework for predicting intracranial aneurysm occlusion outcomes after endovascular treatment. Data from 340 patients include angiographic parametric imaging (API) features, demographic/clinical variables, and aneurysm morphology. Separate deep neural networks are trained on quantitative and categorical data, with late and intermediate fusion schemes, and several class-imbalance augmentation methods (SMOTE/SMOTENC, RandomOverSampler, ADASYN) are compared. Performance is reported as AUROC means and standard deviations over 20-split Monte Carlo cross-validation. The central claim is that SMOTE/SMOTENC augmentation raises intermediate-fusion AUROC from 0.68±0.1 to 0.85±0.06, the highest observed value.","tokens_in":5045,"tokens_out":3586,"duration_ms":36613,"significance":"If the reported gain is genuine, the paper would provide a useful, clinically relevant demonstration that a specific oversampling strategy can substantially improve multimodal deep learning predictions on a small, imbalanced neurovascular dataset. The work compares multiple augmentation techniques on a real 340-patient cohort, which is a practical strength. However, the central result rests on two currently unsupported pillars: the correct temporal placement of augmentation relative to cross-validation splits, and statistical evidence that the AUROC improvement is not due to chance or leakage. Because these pillars are not established, the significance of the finding cannot yet be assessed.","major_comments":[{"comment":"The manuscript does not state whether SMOTE/SMOTENC, RandomOverSampler, and ADASYN are applied before or after the 20-split Monte Carlo cross-validation splits. This is load-bearing: if augmentation is applied to the entire dataset before splitting, synthetic samples can appear in both training and validation/test folds, and the reported AUROC of 0.85±0.06 for intermediate fusion could be artificially inflated by memorization of synthetic data. Please specify explicitly that the oversampling procedure is fitted on each training fold only and applied to that fold, with no synthetic example entering the corresponding validation or test fold. Also report the minority/majority class counts in the original dataset and the number of synthetic samples generated per method.","section":"Section 2.3 and Figure 1"},{"comment":"The claim that SMOTE/SMOTENC \"significantly enhanced\" the intermediate-fusion model is based on a comparison of 0.68±0.1 versus 0.85±0.06. The paper reports only means and standard deviations over 20 splits, with no confidence intervals, no paired significance test, and no statement about whether the splits are shared across compared methods or whether random seeds are fixed. Given the small cohort (n=340) and the large reported standard deviations, the observed gain may not be statistically reliable. Provide confidence intervals for the AUROC of each configuration and a paired test (e.g., DeLong test or bootstrap) comparing the best augmentation against no augmentation, with the same split indices used across all methods.","section":"Section 3, Results"},{"comment":"The neural network architecture descriptions are too underspecified to reproduce the experiments or to attribute performance differences to fusion type rather than to arbitrary hyperparameter choices. The number of layers, layer widths, L1/L2 regularization coefficients, learning rate values and schedules, early-stopping criteria, and the exact \"advanced pipeline setup\" for the intermediate fusion model are not given. Please provide a complete specification of both architectures, or a pointer to released code, so that the comparison between late and intermediate fusion is interpretable and reproducible.","section":"Section 2.2, Data fusion"}],"minor_comments":[{"comment":"The abbreviation \"SMOTE NC\" appears with a space in the abstract, while the main text uses \"SMOTENC\"; please use one consistent form throughout.","section":"Abstract"},{"comment":"The phrase \"pre decision layers\" should be hyphenated as \"pre-decision layers\" (it also appears in the abstract and the summary).","section":"Section 2.2"},{"comment":"The results are reported only in narrative form; a table summarizing AUROC (mean±std) for each fusion type and each augmentation method would greatly improve readability and allow direct comparison.","section":"Section 3"},{"comment":"The two-step normalization method attributed to Ionita et al. is described only by a reference to [7]; the two steps themselves are not described. Please provide a brief description or cite the original methodological source explicitly.","section":"Section 2.1"},{"comment":"There is no data or code availability statement. Given the reproducibility concerns, please add a sentence indicating whether the de-identified dataset and implementation can be shared.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually brief for the claims it makes, and the main risk is the unspecified temporal relationship between oversampling and cross-validation splitting. I would ask the editor to ensure the authors answer the leakage question directly; if the augmentation was correctly performed per fold, the result may stand, but the absence of this detail currently prevents validation. The paper also lacks any significance testing, which is surprising given the repeated \"significant\" language in the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper reports a large AUROC jump from 0.68 to 0.85 when SMOTE/SMOTENC is applied to an intermediate fusion model for aneurysm occlusion, using 340 patients. The contribution is modest but real: it applies standard augmentation strategies to a multimodal clinical dataset that wasn't covered in the prior API-only work (ref [7]). That said, the central result isn't yet trustworthy, because the paper never states whether augmentation happens inside each cross-validation fold or before splitting. If SMOTENC synthesizes examples from the full dataset, validation folds can contain information from training patients, and the 0.85 number becomes an artifact.\n\nWhat's good: the architecture is described clearly enough to reproduce, the late vs intermediate fusion comparison is sensible, and the paper is honest that other augmentation methods hurt performance. The clinical question is worth asking; interventionalists don't have good occlusion prediction models. Combining API with demographics and morphology is a reasonable extension of ref [7].\n\nSoft spots: (1) augmentation/CV timing is the big one; it's the difference between a real finding and leakage. Figure 1 shows augmentation after preprocessing and before training, which is ambiguous. (2) No confidence intervals or significance tests are given, and with 20 Monte Carlo splits those are cheap to compute; the spread across methods (AUROC 0.52 to 0.85) suggests high variance. (3) No code or data is provided, and the minority class size isn't reported, so the effective sample is unknown. (4) The unimodal AUROC is 0.74 while the multimodal baseline is 0.68; the authors don't discuss why adding morphology and demographics hurts, which matters for their claim that fusion helps.\n\nI'd send this to a serious referee. The methodological ambiguity is exactly what peer review should catch, and the dataset and clinical question are valuable. But I wouldn't cite the 0.85 result until the leakage question is answered. For a reading group it's a useful case study in augmentation and cross-validation hygiene.","headline":"A clinically relevant multimodal fusion study whose headline AUROC gain to 0.85 depends on an unspecified augmentation/split ordering, so the result is not yet citable.","tokens_in":5555,"tokens_out":1808,"would_cite":false,"duration_ms":17537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An intermediate-fusion neural network combining angiographic imaging, biomarkers, and morphology, trained with SMOTE/SMOTENC oversampling, raises aneurysm occlusion prediction AUROC from 0.68±0.1 to 0.85±0.06.","keywords":["Intracranial Aneurysms","Multimodal Machine Learning","Data Augmentation","Predictive Accuracy","SMOTE/SMOTENC","Treatment Outcomes","Angiographic Parametric Imaging","Intermediate Fusion"],"falsifier":"Re-run the 20-split Monte Carlo cross-validation while instrumenting the pipeline to record where each synthetic SMOTE/SMOTENC sample is created relative to the train/validation/test boundaries; if the intermediate fusion AUROC drops from 0.85±0.06 to the un-augmented 0.68±0.1 (or below the 0.74 unimodal baseline) when synthetic samples are confined to training folds, the paper's central claim is refuted.","tokens_in":4588,"feed_emoji":"🧠","tokens_out":10034,"duration_ms":89289,"temperature":0.7,"pith_summary":"Intracranial aneurysm treatment outcomes are hard to predict, and the clinical dataset is imbalanced because successful occlusions far outnumber failures. This paper asks whether combining three data sources — hemodynamic maps from angiographic parametric imaging (API), patient demographics and history, and aneurysm morphology — can beat a single-modality model once that imbalance is handled with data augmentation. On 340 treated patients, the unimodal API deep network achieved an AUROC (area under the receiver operating characteristic curve) of 0.74±0.07, while un-augmented late and intermediate fusion dropped to 0.68±0.1. Adding SMOTE/SMOTENC oversampling raised the intermediate fusion model to 0.85±0.06, the best result in the study, and improved late fusion to 0.73±0.04. If this holds under properly separated augmentation, the framework offers a data-driven way to forecast which aneurysms may not occlude, using information already collected in routine care.","feed_headline":"Fusing data plus oversampling pushes aneurysm prediction AUROC to 0.85","feed_subtitle":"Fusing imaging, biomarkers, and morphology with SMOTE-NC oversampling lifts the best model to AUROC 0.85.","key_machinery":"The central mechanism is the intermediate fusion deep network: separate dense branches process continuous imaging features, continuous numerical features, and one-hot encoded categorical features, and their pre-decision layer outputs are concatenated before passing through additional dense layers to a sigmoid output. The companion mechanism is SMOTE/SMOTENC, a synthetic minority oversampling technique that creates new training examples for the under-represented occlusion-failure class by interpolating between nearest neighbors, with the SMOTENC variant explicitly handling mixed nominal and continuous data. Together they allow the model to learn from a balanced representation of both outcome classes instead of being dominated by the majority successful-occlusion class.","core_discovery":"The paper's central finding is that the way multimodal information is merged and the way class imbalance is handled jointly determine whether fusion helps. Intermediate fusion — where each data type passes through its own short neural network and the learned representations are concatenated before the final prediction layers — paired with SMOTE/SMOTENC synthetic oversampling achieves AUROC 0.85±0.06, the highest performance observed, compared with 0.68±0.1 for the same fusion without augmentation. Late fusion with the same augmentation reaches 0.73±0.04, while random oversampling and ADASYN produce widely varying results, with AUROC between 0.52±0.07 and 0.79±0.1 depending on the architecture. The authors interpret this as evidence that multimodal integration can improve prognostic accuracy provided the augmentation strategy is matched to the architecture and to the mixed categorical-and-continuous feature space.","pith_inferences":["Because the paper does not state whether SMOTE is applied before or after the train/test split, the strongest check is to rerun the same 20-split Monte Carlo protocol while generating synthetic samples only inside each training fold; if validation AUROC falls back toward the un-augmented 0.68–0.74 range, the reported gain is an artifact of leakage.","The two-step normalization of API values for injection variability and foreshortening likely carries part of the benefit, so ablating that preprocessing step would reveal how much of the 0.85 AUROC depends on it.","The model's clinical utility hinges on sensitivity for the minority class, the non-occluded aneurysms; reporting precision and recall for that class would be a more decision-relevant endpoint than AUROC alone.","The same intermediate-fusion-plus-SMOTENC recipe is portable to other imbalanced medical prediction tasks where continuous measurements and categorical risk factors coexist."],"forward_implications":["Intermediate fusion plus SMOTE/SMOTENC is the strongest configuration studied, reaching AUROC 0.85±0.06 versus 0.68±0.1 without augmentation.","Late fusion with the same augmentation reaches AUROC 0.73±0.04, so the gain is not generic to any fusion architecture.","Unimodal API prediction (0.74±0.07) still beats un-augmented multimodal fusion, implying that naive fusion of extra variables can degrade accuracy.","Augmentation choice matters: random oversampling and ADASYN produced AUROC from 0.52±0.07 to 0.79±0.1 depending on the architecture.","If the result generalizes, the framework turns routine angiography, demographic, and morphological data into an occlusion-probability estimate that could inform endovascular treatment planning."],"supporting_citations":[{"why":"Establishes the preceding API-plus-deep-neural-network approach for aneurysm occlusion prediction that this study extends to multimodal fusion.","marker":"[7]"},{"why":"Provides the time-density curve quantitative flow analysis method underlying the API parameters.","marker":"[5]"},{"why":"Supplies the time-density curve and intra-aneurysmal velocity evaluation that the API features draw on.","marker":"[6]"},{"why":"Makes the clinical stakes of subarachnoid hemorrhage from intracranial aneurysms explicit, motivating outcome prediction.","marker":"[1]"},{"why":"Documents the detection and management context for unruptured intracranial aneurysms that frames the prognostic goal.","marker":"[2]"}],"fun_headline_variants":["Multimodal fusion plus oversampling lifts aneurysm AUROC to 0.85","Data fusion and SMOTE-NC boost aneurysm outcome predictions to 0.85 AUROC","How you fuse and oversample decides aneurysm prediction gains","Augmentation strategy keys aneurysm prediction: intermediate fusion hits 0.85","Imbalance beating via SMOTE-NC boosts fused aneurysm model to 0.85"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"In Section 2.3 the augmentation step is described, but the paper never states whether SMOTE/SMOTENC synthetic samples are generated inside each training fold or before the data split, and the 0.85 AUROC claim depends on synthetic examples not leaking into validation or test folds.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal fusion plus oversampling lifts aneurysm AUROC to 0.85","Data fusion and SMOTE-NC boost aneurysm outcome predictions to 0.85 AUROC","How you fuse and oversample decides aneurysm prediction gains","Augmentation strategy keys aneurysm prediction: intermediate fusion hits 0.85","Imbalance beating via SMOTE-NC boosts fused aneurysm model to 0.85"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3308,"prompt_tokens":877,"completion_tokens":2431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2330}},"tokens_in":493,"tokens_out":2431,"duration_ms":16235,"temperature":1.0,"reasoning_tokens":2330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:12:05.536472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 20-split Monte Carlo cross-validation while instrumenting the pipeline to record where each synthetic SMOTE/SMOTENC sample is created relative to the train/validation/test boundaries; if the intermediate fusion AUROC drops from 0.85±0.06 to the un-augmented 0.68±0.1 (or below the 0.74 unimodal baseline) when synthetic samples are confined to training folds, the paper's central claim is refuted.","supporting_citations":[{"cited_title":"Feasibility study for use of angiographic parametric imaging and deep neural networks for intracranial aneurysm occlusion prediction,","cited_arxiv_id":null,"evidence_quote":"Establishes the preceding API-plus-deep-neural-network approach for aneurysm occlusion prediction that this study extends to multimodal fusion."},{"cited_title":"Peritherapeutic quantitative flow analysis of arteriovenous malformation on digital subtraction angiography,","cited_arxiv_id":null,"evidence_quote":"Provides the time-density curve quantitative flow analysis method underlying the API parameters."},{"cited_title":"Evaluation of intraaneurysmal blood velocity by time -density curve analysis and digital subtraction angiography,","cited_arxiv_id":null,"evidence_quote":"Supplies the time-density curve and intra-aneurysmal velocity evaluation that the API features draw on."},{"cited_title":"Unruptured intracranial aneurysms: An updated review of current concepts for risk factors, detection and management,","cited_arxiv_id":null,"evidence_quote":"Makes the clinical stakes of subarachnoid hemorrhage from intracranial aneurysms explicit, motivating outcome prediction."}],"review_version":1}