{"id":"6ae9ae99-2859-403b-b276-69e906647f25","arxiv_id":"1908.06948","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 500-patient public ultrasound dataset and benchmark shows encoder-decoder networks match expert-level left ventricle volume estimation (correlation 0.95) but only moderate ejection fraction accuracy (correlation 0.80).","lead":"This paper introduces CAMUS, a public dataset of ultrasound heart videos from 500 patients with expert-drawn outlines of the left ventricle and left atrium. It shows that deep neural networks trace these outlines and estimate heart volumes almost as well as human experts, while ejection fraction estimates remain less reliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.95/9.5 ml volume and 0.80/5.6% EF figures are computed against one expert (O1a) and ED/ES frames chosen by LV size extremes without ECG (Sec. III-B-2), which the paper calls inaccurate in abnormalities; if O1a is biased, the headline clinical claim overstates accuracy against a true standard.","rationale":"The paper's evaluation is mostly solid: 10-fold cross-validation, a public dataset, inter/intra-observer baselines, and separate reporting of poor-quality results provide real support for the segmentation-level claims. My concern is with the clinical headline, which is anchored to a single expert and to a frame-selection protocol the paper itself flags as not the most accurate. The training procedure makes this dependence concrete: models are trained on O1a contours and evaluated on O1a contours; Fig. 4 shows expert-specific training helps, and Table IV shows experts disagree substantially (LVEF correlation between 0.57 and 0.80). The authors' own caveat in Section III-B-2 is a stated limitation rather than a hidden flaw; I weigh it as a condition on the central clinical claim, not as a reason to reject. The proposed check with O2/O3 on fold 5 is feasible with existing annotations and would show whether the 0.95/9.5 ml and 0.80/5.6% numbers are expert-general or O1a-specific. Because this is the same weakest assumption the reader identified and the verdict is already CONDITIONAL, I do not propose changing the verdict.","tokens_in":33103,"tokens_out":8240,"duration_ms":74035,"concrete_test":"Recompute U-Net 2's clinical indices on fold 5 (40 good/medium patients) using O2 and O3 as reference instead of O1a, and compare the correlations and MAEs to the inter-observer rows of Table IV. If the EDN-vs-O2/O3 LVEF correlation falls below the O1a-vs-O2/O3 range (0.65-0.80) or the volume MAE approaches or exceeds the inter-observer range (about 19-39 ml), the 'faithfully reproduce expert analysis' headline is specific to O1a rather than to a consensus clinical standard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EDNs 'faithfully reproduce the expert analysis' for LV volumes is measured entirely against the annotations of a single cardiologist, O1a, and against ED/ES instants selected as the frames where the LV dimension is largest/smallest, with no ECG gating. Section III-B-2 states this is 'not the most accurate way, especially in the presence of abnormalities' and that clinical indices 'have to be interpreted with this in mind.' The model is trained and evaluated on O1a, so high agreement (corr 0.95, MAE 9.5 ml for volumes; corr 0.80, MAE 5.6% for EF) establishes that EDNs can learn O1a's contouring and frame-selection behavior, not that the resulting clinical indices are accurate with respect to a guideline-defined standard. Table IV shows how much the reference expert matters: O1a-vs-O3 correlations are 0.895 (EDV), 0.860 (ESV) and 0.646 (EF), and O2-vs-O3 EF correlation is only 0.569. Since 49% of the dataset has EF ≤ 45%, the frame-selection limitation applies to a large fraction of pathological cases. The paper's own Fig. 4 confirms that a model trained on one expert performs best when tested against that same expert, so the reported numbers should be read as expert-specific fidelity rather than general clinical accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CAMUS, a publicly available dataset of 500 patients with 2D four- and two-chamber echocardiographic sequences, manual annotations of the left ventricular endocardium, epicardium, and left atrium by one cardiologist on the full dataset and by three cardiologists on a 50-patient fold, plus repeated annotations for intra-observer analysis. The authors benchmark five encoder-decoder networks (two U-Net variants, ACNN, SHG, U-Net++) against two non-deep-learning baselines (SRF and BEASM, in fully- and semi-automatic versions) using 10-fold cross-validation. They report that the deep networks outperform the non-deep baselines on geometric metrics, that U-Net offers the best accuracy/parameter trade-off, that training on more than about 250 patients yields diminishing returns, and that the deep models estimate end-diastolic and end-systolic volumes with high correlation (0.95) and low absolute error (9.5 ml) relative to the reference expert, while ejection fraction estimates show lower correlation (0.80) and MAE 5.6%. The paper also analyzes the effect of image quality, multi-structure training, and expert-specific annotation behavior.","tokens_in":33353,"tokens_out":3247,"duration_ms":33019,"significance":"The CAMUS dataset is a substantial community resource: it is the largest publicly available fully annotated 2D echocardiography dataset, with a rigorous annotation protocol, inter- and intra-observer variability measurements, and a public benchmark platform. The evaluation is unusually thorough for the field: 10-fold cross-validation, strict Wilcoxon tests, additional experiments on training set size, image quality, and expert-specific models, plus extensive supplementary material including Bland-Altman plots and outlier analyses. If the results are taken as stated, the conclusion that simple U-Net architectures match or exceed more complex encoder-decoder designs on this task is a useful, reproducible finding. The study also provides concrete evidence on the number of training patients needed (roughly 200-250) and on the limits of current deep models for ejection fraction estimation, which is a clinically important and honest negative result.","major_comments":[{"comment":"The headline clinical numbers (correlation 0.95, MAE 9.5 ml for volumes; correlation 0.80, MAE 5.6% for EF) are all computed against a single reference expert, O1a, with ED/ES frames chosen as the frames of largest/smallest LV dimension without ECG gating. As the authors themselves state in Section III-B-2, this frame-selection strategy is 'not the most accurate way, especially in the presence of abnormalities,' and Table IV shows very large inter-observer differences (e.g., O2-vs-O3 LVEF correlation of 0.569). The abstract's statement that EDNs 'faithfully reproduce the expert analysis' is therefore accurate only as a statement of fidelity to O1a's contouring and frame-choice behavior, not as clinical accuracy against a guideline-defined standard. I request that the abstract and conclusions be reworded to make this distinction explicit, and that the authors add, at least for the 50-patient multi-expert fold, a sensitivity analysis reporting clinical metrics against O2 and O3 as alternative references.","section":"III-B-2 and Table IV"},{"comment":"All primary segmentation and clinical metrics in the main tables exclude the 19% poor-quality images (94 patients), as stated in Section V. The abstract, however, reports the volume correlation and MAE without this caveat, which could mislead readers into thinking the numbers apply to the full 500-patient dataset. The supplementary Table IX does provide poor-quality results, but I recommend stating the exclusion explicitly in the abstract and reporting the combined or poor-quality results in the main text, since generalization to real clinical data depends on performance across all image qualities.","section":"Section V (before Table III) and abstract"},{"comment":"The claim that 'EDN results are better than the inter-observer scores, on all structures and metrics' compares EDN metrics computed on 406 patients with inter-observer metrics computed on only 40 patients from fold 5. This is not an apples-to-apples comparison: the sample sizes, case mix, and annotation conditions differ. Supplementary Table VII does provide EDN metrics on the same 40-patient fold, and the discussion should base its inter-/intra-observer comparison on those numbers rather than on the full-dataset scores, to avoid a potentially misleading comparison.","section":"V-D-5"}],"minor_comments":[{"comment":"The phrase 'Although these results are below the inter-observer scores' in the abstract is ambiguous, since 'below' could mean worse; the intended meaning is that the errors are smaller than inter-observer errors. Please rephrase (e.g., 'are better than the inter-observer scores').","section":"Abstract and Section I"},{"comment":"The intra-observer row (O1a vs O1b) reports an LVEF MAE of 0.9%, which seems implausibly low compared to the other intra-observer values and to the reported correlation of 0.896. Please verify this entry, since it appears in both Table IV and Table VIII in the supplementary material.","section":"Table IV footnote"},{"comment":"The description of U-Net++ states that 'the original version had 9M parameters but we adapted it,' but the total parameter count in the text (1.1M) is not reconciled with the architectural changes described. A brief sentence explaining the reduction (e.g., reduced feature maps) would improve reproducibility.","section":"Section IV-A-4"},{"comment":"The caption says 'Segmentation accuracy for ACNN architecture' but the table reports U-Net++ variants; this is a typo and should be corrected.","section":"Supplementary Table VI"}],"recommendation":"major_revision","confidential_remarks":"The dataset release is genuinely valuable and the evaluation is much more rigorous than typical for this area. The central issue is not the methodology but the framing: the clinical-accuracy claims are only validated against a single expert, and the paper's own data show that expert choice has a large effect. This is fixable with rewording and a sensitivity analysis, so I do not recommend rejection. The authors should also double-check the apparent inconsistency in the intra-observer LVEF MAE in Table IV."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real dataset contribution and a careful benchmark, and it deserves peer review. CAMUS fills a genuine gap—the largest public fully annotated 2D echo dataset at the time, with 500 patients, two views, three structures, and inter/intra-observer folds. The evaluation is unusually thorough: 10-fold cross-validation, Wilcoxon tests, hyperparameter ablations, poor-quality-image analyses, outlier counts, Bland-Altman plots, and an honest discussion of why fancier architectures don't beat U-Net. The plateau-at-250-patients finding and the robustness-to-image-quality result are useful empirical observations.\n\nThe soft spots are real but manageable. First, the headline clinical numbers (corr 0.95, MAE 9.5 ml; EF corr 0.80, MAE 5.6%) are computed against a single expert, O1a, and against ED/ES frames chosen by LV-size extremes without ECG. The paper acknowledges this in Section III-B-2, but the abstract and conclusions present the numbers without that caveat, and the phrase \"faithfully reproduce the expert analysis\" is doing more work than it should. Table IV shows inter-observer EF correlations as low as 0.57, so the reference-expert choice really matters. This doesn't sink the paper—the dataset is the contribution and the method ranking is robust—but the clinical indices should be read as expert-specific fidelity, not guideline-level accuracy. Second, the abstract's sentence \"Although these results are below the inter-observer scores, they remain slightly worse than the intra-observer's ones\" is contradictory on its face; the discussion uses \"below\" to mean lower error, but for correlations the EDNs are above some inter-observer values. That needs a rewrite. Third, no code is shipped, and some baselines are the authors' own prior work; neither is disqualifying, but code would strengthen reproducibility beyond the public dataset.\n\nThe stress-test concern about O1a anchoring mostly lands, but with nuance: the paper already contains the main caveat in Section III-B-2. The issue is presentation, not a hidden flaw. The central finding—EDNs outperform non-deep methods on heterogeneous clinical data—holds up.\n\nVerdict: solid paper, conditional on revision. I'd send it to serious peer review. It is useful for anyone building or using public echo benchmarks, and I'd bring it to a reading group.","headline":"A genuine dataset-and-benchmark contribution that deserves peer review; the clinical numbers are anchored to one expert and a no-ECG frame choice, and the paper's own wording about inter-observer comparisons needs a cleanup, but the central comparison holds up.","tokens_in":34043,"tokens_out":3730,"would_cite":true,"duration_ms":36223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Encoder-decoder networks, especially U-Net, segment 2D echocardiography at expert-level accuracy, reproduced by the new CAMUS dataset.","keywords":["CAMUS dataset","2D echocardiography","encoder-decoder network","U-Net","cardiac segmentation","left ventricle","ejection fraction","deep learning"],"falsifier":"Re-annotate a subset of the 500 patients using ECG-gated frame selection and a consensus of several cardiologists, then recompute U-Net's volume and ejection fraction errors against that independent reference. If the correlation drops well below 0.95, or if U-Net no longer beats the non-deep methods, the claim that encoder-decoder networks faithfully reproduce expert echocardiographic analysis would be refuted.","tokens_in":32841,"feed_emoji":"🫀","tokens_out":7060,"duration_ms":63016,"temperature":0.7,"pith_summary":"The paper introduces CAMUS, a publicly available dataset of 500 patients' two-chamber and four-chamber 2D echocardiographic acquisitions with cardiologist-annotated contours, and uses it to test how far encoder-decoder convolutional networks go at segmenting the left ventricle endocardium, epicardium, and left atrium and estimating clinical indices. It claims that encoder-decoder networks, especially a compact U-Net, outperform non-deep-learning state-of-the-art methods and produce left ventricular volume estimates that faithfully reproduce expert analysis, with mean correlation 0.95 and mean absolute error 9.5 ml relative to the reference cardiologist. Ejection fraction estimation is more contrasted (correlation 0.80, MAE 5.6%), and segmentation accuracy sits between inter-observer and intra-observer variability. The dataset and the benchmark answer practical questions: how many patients are needed to train such a network (roughly 250 to reach a plateau), and whether more sophisticated architectures help (they do not, for this task).","feed_headline":"U-Net reproduces expert heart-volume readings on 500 echo patients","feed_subtitle":"Encoder-decoder networks beat classical methods and approach expert repeatability on the new 500-patient CAMUS dataset.","key_machinery":"The load-bearing object is the CAMUS dataset itself: 500 patients, two standard views each, manual contours for the left ventricle endocardium, epicardium, and left atrium, plus a dedicated 50-patient fold annotated by three cardiologists and by one cardiologist twice, which defines inter- and intra-observer variability. The algorithm side rests on encoder-decoder networks (EDNs), the U-Net being a convolutional encoder that compresses the image into high-level features followed by a decoder that upsamples back to a segmentation map, with skip connections that recover fine detail. Volumes and ejection fraction are derived from the segmented endocardium using the Simpson biplane method of discs, and all methods are compared through ten-fold cross-validation against the reference expert's annotations.","core_discovery":"The central claim is that encoder-decoder deep networks can be trained, on a few hundred patients, to segment 2D echocardiography at a level that is clinically usable: U-Net's contours and volume estimates land between the agreement of different cardiologists and the repeatability of a single cardiologist. Concretely, U-Net 2 reaches Dice scores around 0.94 to 0.95 for left ventricular structures with mean absolute distances near 1.6 mm, and its end-diastolic and end-systolic volumes correlate at 0.95 with the reference expert at a mean absolute error of 9.5 ml. The paper also establishes that the architectural sophistication of ACNN, stacked hourglasses, or U-Net++ does not beat a well-tuned U-Net, and that training on roughly 250 patients is enough to reach near-peak performance on this dataset.","pith_inferences":["Editorial inference: a natural next test is multi-center, multi-vendor data; this study uses one scanner and one hospital, so the claimed error levels have not been shown to transfer to other equipment.","Editorial inference: because end-diastole and end-systole were picked by largest and smallest left-ventricular dimension rather than by ECG, part of the ejection fraction error may reflect frame-timing in the reference itself; an ECG-gated re-reading of the same patients would separate segmentation error from timing error.","Editorial inference: the finding that multi-structure training does not help left-ventricle segmentation suggests context is not being exploited; a network that explicitly models the relationship between endocardium, epicardium, and atrium could potentially surpass the U-Net plateau.","Editorial inference: the observed outlier rate of about 18%, close to the expert's own re-annotation rate of 13%, points to a concrete goal: data augmentation that reproduces unusual probe tilts, zoom, and shadowing, which the paper's error analysis identifies as the main causes of failures."],"forward_implications":["Fully automatic U-Net contours can supply left-ventricular volume measurements in routine echo reading without a human tracing the endocardium each time.","The ejection fraction estimate, though not yet at intra-observer repeatability, is closer to experts than to classical non-deep methods; adding temporal coherence across the cardiac cycle is the stated next step to improve it.","A compact U-Net (about 2 million parameters) gives essentially the same accuracy as much larger or structurally more complex networks, so clinical deployment does not need expensive architectures.","Training on 250 patients is a practical target for building new echo segmentation systems; adding more patients still helps the most difficult structures but with diminishing returns.","CAMUS provides a public benchmark where future methods can be measured against the same inter- and intra-observer variability."],"supporting_citations":[{"why":"Supplies the U-Net encoder-decoder architecture used as the primary segmentation model.","marker":"[18]"},{"why":"Provides the anatomically constrained neural network (ACNN) baseline and the shape-regularization approach tested against U-Net.","marker":"[19]"},{"why":"Defines the prior standardized echocardiography segmentation benchmark and evaluation protocol that this work extends.","marker":"[3]"},{"why":"Delivers the shape-prior B-spline active surface method that is the main modern non-deep-learning reference.","marker":"[5]"},{"why":"Describes the Simpson biplane method used to convert endocardial contours into volume and ejection fraction measurements.","marker":"[21]"},{"why":"Establishes the chamber-quantification and contouring protocol that defines the ground-truth annotations.","marker":"[22]"},{"why":"Shows that a U-Net can segment the left ventricle in 2D ultrasound, the direct precursor to this study.","marker":"[17]"},{"why":"Describes the structured random forest implementation used as the learning-based non-deep baseline.","marker":"[26]"},{"why":"Supplies the stacked hourglass architecture used as one of the sophisticated encoder-decoder baselines.","marker":"[24]"},{"why":"Supplies the nested U-Net++ architecture used as another sophisticated encoder-decoder baseline.","marker":"[25]"}],"fun_headline_variants":["U-Net matches expert echo readings on new 500-patient dataset","New 500-patient echo dataset: U-Net beats classical methods","Deep learning segments echo hearts: open dataset benchmarks","U-Net approaches expert repeatability on echocardiography volumes","Largest open echo dataset: deep nets near expert volume readings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers are all measured against the manual contours and frame choices of a single cardiologist (O1a), with end-diastole and end-systole picked as the frames where the left ventricle looks largest and smallest rather than by ECG; if that reference is systematically off, the reported correlations and errors overstate how accurate the networks truly are.","fun_headline_variants_meta":{"raw":{"variants":["U-Net matches expert echo readings on new 500-patient dataset","New 500-patient echo dataset: U-Net beats classical methods","Deep learning segments echo hearts: open dataset benchmarks","U-Net approaches expert repeatability on echocardiography volumes","Largest open echo dataset: deep nets near expert volume readings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2306,"prompt_tokens":1025,"completion_tokens":1281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1197}},"tokens_in":641,"tokens_out":1281,"duration_ms":9520,"temperature":1.0,"reasoning_tokens":1197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:02:35.266982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a subset of the 500 patients using ECG-gated frame selection and a consensus of several cardiologists, then recompute U-Net's volume and ejection fraction errors against that independent reference. If the correlation drops well below 0.95, or if U-Net no longer beats the non-deep methods, the claim that encoder-decoder networks faithfully reproduce expert echocardiographic analysis would be refuted.","supporting_citations":[{"cited_title":"U-Net: Convolutional Net- works for Biomedical Image Segmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the U-Net encoder-decoder architecture used as the primary segmentation model."},{"cited_title":"Anatomically Constrained Neural Net- works (ACNNs): Application to Cardiac Image Enhancement and Seg- mentation,","cited_arxiv_id":null,"evidence_quote":"Provides the anatomically constrained neural network (ACNN) baseline and the shape-regularization approach tested against U-Net."},{"cited_title":"Standard- ized Evaluation System for Left Ventricular Segmentation Algorithms in 3D Echocardiography,","cited_arxiv_id":null,"evidence_quote":"Defines the prior standardized echocardiography segmentation benchmark and evaluation protocol that this work extends."},{"cited_title":"Fast and Fully Automatic Left Ventricular Segmenta- tion and Tracking in Echocardiography Using Shape-Based B-Spline Explicit Active Surfaces,","cited_arxiv_id":null,"evidence_quote":"Delivers the shape-prior B-spline active surface method that is the main modern non-deep-learning reference."},{"cited_title":"Assessment of left ventricular ejection fraction and volumes by real-time, two-dimensional echocardiography. A comparison of cineangiographic and radionuclide techniques,","cited_arxiv_id":null,"evidence_quote":"Describes the Simpson biplane method used to convert endocardial contours into volume and ejection fraction measurements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the chamber-quantification and contouring protocol that defines the ground-truth annotations."},{"cited_title":"2D left ventricle segmentation using deep learning,","cited_arxiv_id":null,"evidence_quote":"Shows that a U-Net can segment the left ventricle in 2D ultrasound, the direct precursor to this study."},{"cited_title":"A fully auto- matic and multi-structural segmentation of the left ventricle and the myocardium on highly heterogeneous 2D echocardiographic data,","cited_arxiv_id":null,"evidence_quote":"Describes the structured random forest implementation used as the learning-based non-deep baseline."},{"cited_title":"Stacked hourglass networks for human pose estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the stacked hourglass architecture used as one of the sophisticated encoder-decoder baselines."},{"cited_title":"Unet++: A nested u-net architecture for medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the nested U-Net++ architecture used as another sophisticated encoder-decoder baseline."}],"review_version":1}