{"id":"88963e5b-b36a-47b3-97a2-69c0d212f945","arxiv_id":"1908.04181","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A combination of ImageNet-pretrained 2D/3D CNNs, segmentation-based regularization, and ensemble selection achieves low cross-validation errors on LVQuan19 cardiac MRI, but test-set performance is much worse.","lead":"This paper trains pretrained 2D and 3D convolutional networks to predict left ventricle measurements from cardiac MRI, adding a segmentation-based training regularizer and an ensemble search. The method reports good cross-validation accuracy on the LVQuan19 challenge data but degrades sharply on the held-out test set, so the practical gains are uncertain.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ensemble-search validation is biased: the top-20 candidate pool is ranked on full CV predictions, so the second-portion split used to evaluate the ensemble is not clean; the reported robust gain may be an artifact.","rationale":"The reader's weakest assumption concerns the gap between cross-validation results and test-set results. That gap is real but explicitly acknowledged by the authors in the discussion, and the abstract only claims performance on the training dataset with 5-fold cross-validation. My concern is more specific and more directly tied to the paper's methodological contribution: the ensemble-search validation is internally biased. The starred rows are intended to show that the ensemble-selection gain is not overfit to the CV folds, but the pool selection using full CV predictions contaminates the evaluation split. This is a load-bearing flaw because the ensemble search is advertised in the abstract and is one of the paper's primary contributions. The other contributions (pretraining, segmentation regularization, 2D-to-3D initialization) have independent support in the same table, though also based on the small 56-patient CV. The paper is otherwise transparent about limitations, including the test-set drop. Because the bias is addressable by a corrected evaluation protocol or by reframing the claims, CONDITIONAL remains the appropriate verdict: the paper should either fix the ensemble-evaluation protocol or soften the claims about the ensemble search. I do not see grounds for REJECT, since the main experimental results are internally consistent and the flaw is contained to one component's validation.","tokens_in":8238,"tokens_out":6641,"duration_ms":66478,"concrete_test":"Re-run the ensemble meta-evaluation with the candidate pool S restricted to the top 20 configurations ranked on the first portion of each split only, with the optimal-subset search on the first portion and evaluation on the second portion. Compare the resulting area MAE gain over averaging to Table 1's Optimal* vs. Average* (111 vs 117 mm²). If the gain disappears or fails the Wilcoxon signed-rank test at α=5%, the ensembling claim is an artifact of selection leakage; if the gain persists, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract's central claim rests on the ensemble-search scheme, but the validation of that scheme in §2.2 has a selection-leakage flaw. The authors rank all configurations by individual CV performance to form the top-20 pool S, using the concatenated out-of-fold predictions across all CV folds. They then split each fold into two portions, use the first portion to select the optimal subset, and use the second portion to evaluate the strategy (the starred rows in Table 1). Because the ranking of the pool is computed over the full out-of-fold predictions, the second (evaluation) portion has already influenced which configurations enter S. A configuration that happens to perform well on the evaluation portion is more likely to be included in the pool, and then can be selected. Consequently, 'Ensemble Optimal*' (111±75 mm² for area vs. 117±75 for Average*) is not an unbiased estimate of the ensemble-selection strategy, and the Wilcoxon test performed on that split does not establish that the search scheme generalizes. This directly undermines the robustness emphasis of the abstract and the claim that the ensemble search improves over simple averaging. The CV-to-test gap noted by the reader is acknowledged and discussed by the authors, but this internal selection-leakage is a correctness risk in the paper's own evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a method for automatic left ventricle (LV) quantification from cardiac MRI in the setting of the small LVQuan19 challenge dataset. The method combines pretrained 2D CNNs and 3D spatio-temporal CNNs initialized from ImageNet weights, an architecture-independent segmentation decoder used as a regularizer, and a search scheme that selects an optimal ensemble of trained configurations for each regression/classification task. The authors report patient-level 5-fold cross-validation results on the 56-patient training set, with mean absolute errors of 111 ± 76 mm² for area, 1.84 ± 0.9 mm for dimensions, 1.22 ± 0.6 mm for regional wall thickness, and a 6.7% cardiac phase classification error for the selected ensemble. They also report LVQuan19 test-set results, which are substantially worse, and compare against a multi-task learning baseline.","tokens_in":8485,"tokens_out":6860,"duration_ms":75365,"significance":"If the cross-validation results are taken at face value, the paper makes several useful empirical contributions: it demonstrates that ImageNet transfer learning helps substantially in a very small cardiac MRI dataset, that segmentation-based regularization can improve direct regression, and that 3D spatio-temporal variants initialized from 2D weights are competitive with 2D models. The ensemble-selection idea is relevant to challenge settings, and the authors make an explicit effort to evaluate selection overfitting with an additional split. The experimental design is mostly careful, including patient-level 5-fold CV, Wilcoxon signed-rank tests, and a broad comparison of architectures. However, the central claim that the ensemble search robustly improves over simple averaging is weakened by a selection-leakage flaw in the evaluation protocol, and the statistical evidence for the improvement is incomplete. The test-set degradation is honestly acknowledged but leaves the external validity of the method open.","major_comments":[{"comment":"The additional test split does not remove the selection leakage from the ensemble evaluation. The pool S of top-20 configurations is ranked using the concatenated out-of-fold predictions ŷ_i = ∪_j ŷ_ij, which include the second (evaluation) portion of each fold. Consequently, a configuration that performs well on the evaluation portion by chance is more likely to enter S, and then can be selected by the subset search run on the first portion. Therefore the 'Ensemble Optimal*' row in Table 1 is not an unbiased estimate of the ensemble-selection strategy, and the Wilcoxon test comparing 'Ensemble Optimal*' with 'Ensemble Average*' does not establish that the search scheme generalizes. Please redo this evaluation by ranking the pool using only the first-portion predictions from each fold, or by performing a fully nested cross-validation in which both pool ranking and subset selection are done inside the training folds only.","section":"§2.2 (Ensembling) and Table 1 (starred rows)"},{"comment":"The claim that the optimal ensemble is significantly better than averaging is load-bearing for the paper's main contribution, but no p-values, test statistics, or effect sizes are reported for the comparisons between 'Ensemble Average' and 'Ensemble Optimal' or for the starred split evaluations. The multiple comparisons across the four tasks also warrant correction or at least explicit discussion. Please provide the full test results for both the original CV and the split evaluation, or remove the significance claims.","section":"§3 (Results)"},{"comment":"The test-set results (371 mm² for area, 3.02 mm for dimensions, 2.53 mm for RWT, 11.5% phase error) are two to three times worse than the cross-validation estimates, which directly undercuts the abstract's emphasis on improving robustness through the proposed approach. The discussion acknowledges the gap, but it does not provide an analysis of whether the degradation is due to the ensemble search, the transfer-learning strategy, or a fundamental distribution shift. Please either temper the abstract's robustness claim or add a per-patient and per-fold analysis of the test-set errors that would allow the reader to attribute the gap.","section":"Table 1 and Discussion"}],"minor_comments":[{"comment":"There are several typos: 'performce' should be 'performance', 'averging' should be 'averaging', and 'statically significant' should be 'statistically significant'. In §4, 'statisticially' should also be corrected.","section":"§3"},{"comment":"The number of configurations in the full set S and the relationship between the top-20 pool and the full configuration space are not stated. Please report these numbers, as they affect the computational cost and the interpretation of the ensemble-search bias.","section":"§2.2 (Ensembling)"},{"comment":"For the starred rows, the number of patients in the first and second portions of the additional split is not given. Please state the split sizes so the reader can judge the stability of the ensemble-selection results.","section":"Table 1"},{"comment":"The manuscript does not state whether code or trained models will be released. Given the small dataset and the complexity of the ensemble search, providing code or detailed prediction files would substantially improve reproducibility.","section":"All"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid challenge-style contribution, and the core ideas (transfer learning, segmentation regularization, and 2D/3D ensembling) are reasonable. The main problem is the selection leakage in the ensemble evaluation, which is fixable by re-running the analysis with a clean nested split. The test-set gap is concerning but honestly discussed; I would not reject on that basis alone. Please ask the authors to provide the corrected evaluation and full statistical details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: This is a reasonable LVQuan19 challenge write-up. The transfer-learning results are believable and genuinely useful for small cardiac MRI datasets. The ensemble-search validation, though, has a selection-bias problem that the authors did not fully close, and the test-set numbers are far worse than the CV numbers, which the authors admit but do not resolve.\n\nWhat's new: The architecture-independent segmentation decoder is a small but legitimate addition—adding an auxiliary cross-entropy loss via a decoder before global pooling is simple and effective, and they show significant gains for areas and dimensions. The 2D-to-3D kernel inflation with pretrained ImageNet weights is not new, but for this dataset it works. The comparison across architectures is reasonable, and the patient-level 5-fold CV with Wilcoxon tests is a solid experimental setup.\n\nThe soft spots: (1) The ensemble-search validation in Section 2.2 has leakage. The top-20 pool is ranked on full out-of-fold predictions, then each fold is split in two; the second half is used to evaluate the selected ensemble. But the second half already influenced which configurations entered the pool. So the starred 'Optimal*' rows are not an unbiased estimate of the search scheme. The gap between Optimal* and Average* is small (111 vs 117 mm² area, 1.85 vs 1.96 mm dimension), and the significance test on that split does not establish that the search generalizes. This is a real flaw, not a nitpick. (2) The CV-to-test-set drop is large—area MAE goes from 111 to 371 mm². The authors acknowledge it, but it means the abstract's robustness claim is not supported beyond the training-fold distribution. (3) No code or data artifacts, and the multiple Wilcoxon tests are not corrected for multiplicity. These are minor-to-moderate.\n\nVerdict: The paper is worth a serious referee, especially as a challenge contribution, but it needs revision. Fix the ensemble evaluation (e.g., nested CV for pool selection, or report the bias correctly), reframe the claims around generalization, and ideally release code. I would not cite it as-is for the ensemble method, but the transfer-learning and segmentation-regularization results are citable for small-data cardiac MRI.\n\nRecommendation: Send to peer review, conditional on revision.","headline":"Honest challenge report with solid transfer-learning gains, but the ensemble-search validation leaks selection bias and the CV-to-test drop undercuts the robustness claim.","tokens_in":8997,"tokens_out":4627,"would_cite":false,"duration_ms":41903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that direct regression with pretrained CNNs, segmentation regularization, and ensemble search can quantify left-ventricle indices from a small cardiac MRI dataset, reaching 111 ± 76 mm² area error on cross-validation…","keywords":["left ventricle quantification","cardiac MRI","direct regression","transfer learning","segmentation regularization","ensemble selection","3D CNN","spatio-temporal CNN"],"falsifier":"Run the selected ensemble on a held-out test set of similar cardiac MR images and compare errors: if they stay close to the cross-validation range (area ≈ 111 mm², dimension ≈ 1.84 mm, RWT ≈ 1.22 mm, phase ≈ 6.7%), the claim generalizes; if they reproduce the reported test-set errors (371 mm², 3.02 mm, 2.53 mm, 11.5%), the cross-validation-based claim fails to transfer.","tokens_in":8024,"feed_emoji":"❤️","tokens_out":6958,"duration_ms":66421,"temperature":0.7,"pith_summary":"The paper sets out to show that direct regression with pretrained convolutional networks can quantify the left ventricle from cardiac MRI even when the dataset is very small (56 patients). The method combines ImageNet-pretrained 2D and 3D CNNs, a segmentation-based regularizer that adds a segmentation loss through a decoder branch during training, and an exhaustive search that selects the best ensemble of models for each output task. On the LVQuan19 training set, 5-fold cross-validation gives mean absolute errors of 111 ± 76 mm² for area, 1.84 ± 0.9 mm for dimensions, and 1.22 ± 0.6 mm for regional wall thickness, with 6.7% cardiac-phase classification error. The paper also reports that the same ensemble on the challenge test set gives substantially larger errors (371 mm² area, 3.02 mm dimension, 2.53 mm RWT, 11.5% phase), which it attributes to the test set being more challenging and different from the training distribution.","feed_headline":"Heart-MRI error drops to 111 mm² on cross-validation","feed_subtitle":"Pretrained 2D/3D CNNs plus a segmentation regularizer fit 56 patients; test-set errors stay larger.","key_machinery":"Three devices carry the argument. Pretrained initialization transfers ImageNet image-classification features: every 2D backbone starts from ImageNet weights, and the 3D variants are built by copying each 2D kernel across the temporal depth and multiplying by 1/dc, which makes parameter-heavy 3D spatio-temporal CNNs trainable on 56 patients. Segmentation-based regularization attaches a decoder before global average pooling; a cross-entropy loss on the predicted myocardium and cavity maps backpropagates through the whole network, so the shared representation is pulled toward anatomy while the regression heads read out areas, dimensions, wall thicknesses, and phase. Ensemble search exhaustively evaluates all subsets of the top-20 configurations and selects, per task, the subset whose averaged predictions minimize error on the cross-validation folds, with multi-crop evaluation covering each 300×300 image.","core_discovery":"Under the LVQuan19 challenge conditions, the paper's central claim is that the best way to exploit a small, high-resolution cardiac MRI set is to start from ImageNet-pretrained 2D classifiers, extend selected ones to 3D by copying 2D kernels into 3D kernels and rescaling by 1/dc, force the shared features to also predict myocardium and cavity segmentations through a decoder branch, and then average the task-specific subset of models selected by exhaustive ensemble search. With that recipe, the per-slice predictions are within 111 ± 76 mm² for cavity and myocardium area, 1.84 ± 0.9 mm for the three cavity dimensions, and 1.22 ± 0.6 mm for the six regional wall thicknesses, and the cardiac phase is wrong in 6.7% of slices under cross-validation. The authors are careful to state that the segmentation mask is used only as a regularizer, never as an explicit input at inference, and that the ensemble-selection step was checked on an extra held-out split to guard against overfitting the cross-validation folds. They also report the test-set results explicitly as being substantially worse, concluding that generalizable LV quantification remains open.","pith_inferences":["The same 2D-to-3D weight-copying initialization could be applied to other spatio-temporal medical imaging tasks with scarce annotated volumes, such as cine CMR motion analysis or 4D imaging; the paper demonstrates it only for 20-frame cardiac slice sequences.","Because the segmentation mask is used only as a loss, the method cannot correct segmentation errors at inference; adding a contour-guidance or segmentation-readout path might narrow the training-to-test gap.","The ensemble search selects subsets per task, and the paper does not report the total training cost of the roughly twenty configurations; that cost could be a practical barrier for replication.","The large test-set gap suggests that test-time adaptation or intensity-domain matching to the training distribution may recover part of the lost performance; this is testable with the same models on the challenge test split."],"forward_implications":["Pretrained weights provide the largest single gain: area MAE drops from 199 mm² with random initialization to 139 mm² with ImageNet initialization on DenseNet121 2D.","Adding segmentation regularization improves area and dimension regression significantly over the same architecture without it, even though the predicted segmentation is not used at inference.","3D spatio-temporal CNNs initialized by copying 2D kernels outperform their 2D counterparts on area and dimension regression despite the small dataset.","Selecting a task-specific ensemble subset by exhaustive search beats averaging all models, and the gain persists on a separate evaluation split.","The sizeable drop from cross-validation to test-set performance shows the challenge test set differs from the training distribution, so the practical gain on unseen data is smaller than the cross-validated numbers suggest."],"supporting_citations":[{"why":"Provides the residual block design reused across the ResNet variants evaluated as 2D and 3D backbones.","marker":"[3]"},{"why":"Provides the DenseNet backbones (DN121, DN169, DN161) that form many of the individual models and part of the final ensemble.","marker":"[5]"},{"why":"Supplies ResNeXt-101 (64×4d), the single best-performing architecture in the paper.","marker":"[14]"},{"why":"Supplies the squeeze-and-excitation blocks used in the SE-ResNet and SENet variants.","marker":"[4]"},{"why":"Supports the transfer-learning premise that ImageNet pretraining helps when medical imaging data are scarce.","marker":"[11]"},{"why":"Provides the multi-crop evaluation strategy used to cover the whole image at inference.","marker":"[2]"},{"why":"Offers the prior direct-vs-segmentation comparison this work extends and the observation that previous challenge test sets were also harder.","marker":"[15]"},{"why":"Provides the DMTRL multitask baseline that the paper compares against and the multi-task relationship-learning context for future work.","marker":"[16]"}],"fun_headline_variants":["Pretrained CNNs + segmentation regularizer cut LV error to 111 mm²","Ensemble of pretrained 2D/3D CNNs for LV regression: 111 mm²","LVQuan19: cross-val error 111 mm² area, 1.84 mm dimension","Segmentation as regularizer in cardiac MR yields 111 mm² area error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy rests on the assumption that the 56-patient cross-validation folds are representative of the challenge test distribution, an assumption the paper's own test-set numbers contradict.","fun_headline_variants_meta":{"raw":{"variants":["Pretrained CNNs + segmentation regularizer cut LV error to 111 mm²","Ensemble of pretrained 2D/3D CNNs for LV regression: 111 mm²","LVQuan19: cross-val error 111 mm² area, 1.84 mm dimension","Segmentation as regularizer in cardiac MR yields 111 mm² area error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2186,"prompt_tokens":1042,"completion_tokens":1144,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":1049}},"tokens_in":658,"tokens_out":1144,"duration_ms":11569,"temperature":1.0,"reasoning_tokens":1049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:48:17.229959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the selected ensemble on a held-out test set of similar cardiac MR images and compare errors: if they stay close to the cross-validation range (area ≈ 111 mm², dimension ≈ 1.84 mm, RWT ≈ 1.22 mm, phase ≈ 6.7%), the claim generalizes; if they reproduce the reported test-set errors (371 mm², 3.02 mm, 2.53 mm, 11.5%), the cross-validation-based claim fails to transfer.","supporting_citations":[{"cited_title":"(2016) Deep residual learning for image recog- nition","cited_arxiv_id":null,"evidence_quote":"Provides the residual block design reused across the ResNet variants evaluated as 2D and 3D backbones."},{"cited_title":"(2017) Densely con- nected convolutional networks","cited_arxiv_id":null,"evidence_quote":"Provides the DenseNet backbones (DN121, DN169, DN161) that form many of the individual models and part of the final ensemble."},{"cited_title":"(2017) Aggregated residual trans- formations for deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies ResNeXt-101 (64×4d), the single best-performing architecture in the paper."},{"cited_title":"(2018) Squeeze-and-excitation networks","cited_arxiv_id":null,"evidence_quote":"Supplies the squeeze-and-excitation blocks used in the SE-ResNet and SENet variants."},{"cited_title":"(2016) Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning","cited_arxiv_id":null,"evidence_quote":"Supports the transfer-learning premise that ImageNet pretraining helps when medical imaging data are scarce."},{"cited_title":"(2019) Skin Lesion Classification Using CNNs with Patch- Based Attention and Diagnosis-Guided Loss Weighting","cited_arxiv_id":null,"evidence_quote":"Provides the multi-crop evaluation strategy used to cover the whole image at inference."},{"cited_title":"(2018) Calculation of Anatomical and Functional Metrics Using Deep Learning in Cardiac MRI: Comparison Between Direct and Segmentation-Based Estimation","cited_arxiv_id":null,"evidence_quote":"Offers the prior direct-vs-segmentation comparison this work extends and the observation that previous challenge test sets were also harder."},{"cited_title":"(2018) Full left ventricle quan- tification via deep multitask relationships learning","cited_arxiv_id":null,"evidence_quote":"Provides the DMTRL multitask baseline that the paper compares against and the multi-task relationship-learning context for future work."}],"review_version":1}