{"id":"dbb61338-fa0b-4ad4-8080-b3b9691aff61","arxiv_id":"1908.02582","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Probabilistic segmentation with MC dropout or a Probabilistic U-Net can produce head circumference measurements with sample-based confidence bounds and variance scores, evaluated on 540 fetal ultrasound test images.","lead":"This paper adds uncertainty estimates to an automated deep-learning system that measures fetal head circumference from ultrasound, so a sonographer can see how confident the system is in a given scan view. It tests two probabilistic networks and proposes variance-based scores for rejecting poor-quality images, potentially guiding operators toward better scan planes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Variance-score rejection thresholds are selected and evaluated on the same 540-image test set, so the reported guidance benefit may be pure selection bias.","rationale":"The reader's weakest assumption identifies the same load-bearing risk: the variance scores must generalize to unseen images for the feedback loop to work. My stress-test focuses this into a concrete methodological defect in the supporting experiment: the thresholds that produce the reported rejection benefit are selected and evaluated on the same test set, so the benefit cannot be distinguished from selection bias. This does not overturn the paper's plausible base measurement performance, and the authors do acknowledge limitations, so conditional acceptance remains appropriate. However, without a held-out threshold-selection protocol or external validation, the central claim about real-time feedback guidance is not established. I therefore agree with the reader's conditional verdict and recommend no change.","tokens_in":6810,"tokens_out":2946,"duration_ms":37601,"concrete_test":"Split the 540 test images into a threshold-selection set and a held-out evaluation set (or use an external dataset). On the selection set only, choose the variance-score threshold that maximizes post-rejection MAE improvement, or pre-register a fixed threshold. Then apply that threshold to the held-out set and report MAE, DICE, and rejection rate before and after rejection. If the improvement disappears or reverses on held-out data, the rejection benefit shown in Figure 2 is attributable to in-sample threshold selection rather than to a genuine property of the variance scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central new claim is that variance scores can guide sonographers toward better scan planes by rejecting sub-optimal views. The only quantitative support is the 'Variance Measure Thresholding' experiment (Section 3), where a threshold between 0 and 1 is applied to each variance score over the test set, and the performance of the remaining 'accepted' cases is plotted against the number of rejected cases (Figure 2). No held-out data is used to select these thresholds, and no pre-registered threshold is defined. Because the threshold is chosen after seeing the test-set errors, the initial performance boost from rejecting high-variance cases is expected even if the variance scores carry no information about measurement error: the threshold can exploit idiosyncratic noise in the test set. The paper's own Discussion (Section 4) concedes the scores 'cannot claim' to represent model uncertainty directly, yet the only evidence that they are even a useful view-quality signal comes from this in-sample threshold-fitting exercise. The base HC measurement accuracy (1.81 +/- 1.65 mm) is plausible, but the feedback contribution is the paper's novel element, and it is not supported by transferable evidence. Relatedly, the '50% containment' statistic is not a calibrated confidence interval: N is increased until coverage reaches about 50%, with no predefined confidence level or calibration set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a probabilistic deep-learning pipeline for fetal head circumference (HC) measurement from 2D ultrasound, built on a U-Net segmentation followed by ellipse fitting. Two stochastic segmentation mechanisms are compared: Monte-Carlo Dropout at inference and a Probabilistic U-Net. For each input image the method draws N segmentation samples, fits an ellipse to each sample, and aggregates the resulting ellipse parameters into a point estimate and an upper/lower bound. It also defines four 'variance scores' (ellipse parameter variance, total ring area, mask classification entropy, softmax confidence entropy) and uses them to reject test images above a threshold, with the aim of steering sonographers toward scan planes for which the network is more reliable. Experiments on a dataset of 2848 training and 540 test images report a mean absolute HC error of 1.81 ± 1.65 mm for the best deterministic single-sample setting, approximately 50% containment of ground-truth HC within the predicted bounds at N=1000 MC samples, and improved average performance on accepted images after rejecting high-variance cases in the test set.","tokens_in":7065,"tokens_out":3206,"duration_ms":38083,"significance":"If the central claims were fully validated, the paper would be a useful clinical-technical contribution: real-time confidence bounds and operator feedback for fetal HC measurement could improve inter-operator consistency, and the base measurement accuracy is in line with prior work. The strengths of the paper are its use of a large clinical dataset with external ground-truth annotations, a plausible and well-motivated pipeline, and a comparison between MC dropout and Probabilistic U-Net under a common evaluation protocol. However, the two elements that constitute the paper's novelty—the 'confidence' containment statistic and the variance-score feedback mechanism—are not yet convincingly validated. The containment percentage is not a calibrated confidence statement because the number of samples N is selected after observing the coverage curve, and the variance-score thresholds are selected and evaluated on the same test set. These issues affect the abstract's headline numbers and the real-time feedback claim, so the manuscript needs substantial additional validation before the main contributions can be accepted.","major_comments":[{"comment":"The 'percentage in upper-lower bound range' statistic is not a calibrated confidence interval. The paper reports that coverage varies significantly with N (14.8% at N=2 and 50.4% at N=1000 for MC dropout) and that the 50% figure is reached only after increasing N to 1000. Because N is chosen after observing the coverage curve, the headline '50% of test images fully contained' is an artifact of the sample-count choice rather than a principled confidence statement. To support the confidence claim, the authors should pre-specify a target coverage level, select N (or the interval construction) on a validation set, and only then evaluate coverage on the held-out test set. Alternatively, they should report coverage as a monotone function of N for a fixed N and avoid claiming a calibrated confidence level.","section":"Section 3, Table 2 and Supplementary Figure 4"},{"comment":"The variance-score rejection thresholds are selected and evaluated on the same 540-image test set. For each of the four variance scores, the threshold between 0 and 1 is varied over the test data, and the performance improvement after rejecting high-variance cases is reported on the same images. This is in-sample threshold fitting, so the observed initial performance boost can arise from exploiting idiosyncratic noise in the test set even if the variance scores carry no true signal about measurement error. The correct protocol is to select thresholds on a separate validation set (or via cross-validation) and then report accepted-image performance on an independent test set. Without this, the paper's central feedback claim is not supported.","section":"Section 3, 'Variance Measure Thresholding', and Figure 2"},{"comment":"The authors concede that the proposed variance scores 'cannot claim' to represent model uncertainty directly, but the real-time feedback contribution depends on these scores tracking measurement error or view quality on unseen images. The only quantitative evidence for their utility is the in-sample threshold experiment in Section 3, which is not transferable evidence. The authors should provide a direct analysis relating each variance score to absolute HC error on held-out data (for example, a correlation or calibration plot), and demonstrate that a fixed, pre-selected threshold transfers to a new dataset or at least to an untouched test split. Without such evidence, the claim that the system can guide sonographers toward better scan planes is speculative.","section":"Section 4, Discussion"},{"comment":"The dropout probability p=0.6 is described as 'value found to be best performing in variety of dropout configurations' from single-sample experiments, with results then reported on the Dataset A test data. If the model configuration was selected using the test set, the reported mean absolute error of 1.81 ± 1.65 mm may be optimistic. The authors should state explicitly whether a held-out validation split was used for hyperparameter selection, and if not, they should re-evaluate the final configuration on a truly held-out set. This is a load-bearing point for the base accuracy claim, although the accuracy itself is plausible.","section":"Section 3, 'Single-Sampling Experiments', and Table 1"}],"minor_comments":[{"comment":"The definitions of mask classification entropy and softmax confidence entropy appear to have sign issues: entropy is conventionally written as -∑ p log p, but the equations as printed omit the leading minus sign, and h4's summation index is unclear (the sum over K_i and the definition of p_i are confusing). Please clarify the notation and verify the signs.","section":"Section 2, equations h3 and h4"},{"comment":"The use of 'MC dropout' is ambiguous: Table 1 reports a 'Dropout' model trained with dropout, while Table 2 reports 'MC(inf.)' for dropout applied during inference. Please label clearly whether dropout is applied during training, during inference, or both, and align the terminology across the text and tables.","section":"Section 2 and Table 2"},{"comment":"The horizontal axis shows the number of rejected cases, but the text describes thresholds 'between zero and one'. The figures would be easier to interpret if the threshold value itself were shown on a secondary axis, or if curves were plotted against the normalized threshold for each variance score.","section":"Figure 2"},{"comment":"The reported DICE and Hausdorff distance values are given as mean ± standard deviation, but no confidence intervals or statistical significance tests are provided. Given the small differences between some configurations, a note on variability across runs or a statistical comparison would strengthen the claims.","section":"Tables 1 and 2"},{"comment":"The abstract states '50% of the test images fully contained within the predicted confidence margins' without mentioning that this percentage is specific to N=1000 MC samples and not a calibrated confidence level. Please qualify this statement in the abstract to avoid overclaiming.","section":"Abstract"},{"comment":"Reference 13 is formatted inconsistently with the other entries (listing the institution name as the author). Please standardize the reference style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a short conference-style paper whose base segmentation and measurement results are plausible and useful, but the two novel claims—calibrated confidence bounds and variance-score-based feedback—are under-validated. The in-sample threshold selection and the post-hoc choice of N are the key problems; both are fixable with a held-out validation protocol and more careful evaluation. I would not recommend rejection, but the revision needs to include substantial additional experiments before publication. The authors may also wish to temper the abstract's wording, since the Discussion itself disclaims direct uncertainty interpretation of the variance scores."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Put simply, the paper's base measurement is credible, but its new feedback signal is not yet demonstrated. The authors add MC Dropout and Probabilistic U-Net to their earlier fetal HC pipeline, sample multiple segmentations per image, fit ellipses to each, and derive four 'variance scores' (h1-h4) from the sample spread. The idea is to reject high-variance images during scanning so sonographers get real-time guidance toward better planes. That combination is new in fetal ultrasound, and the clinical motivation is clear.\n\nThe paper does some things well. The dataset is large: 2,724 volunteer exams plus HC18, with images labeled by 45 sonographers. The single-sample result (1.81 ± 1.65 mm mean error) is consistent with the group's earlier work and with human-level performance. The authors are also honest: they explicitly say in Section 4 that the variance scores 'cannot claim' to represent model uncertainty directly, which is more than many papers in this area do. The qualitative examples of low-signal images producing wide margins are useful.\n\nThe soft spots are where the actual claims live. The variance-score rejection experiment in Section 3 sweeps a threshold between 0 and 1 on the test set, rejects the cases above it, and plots the error of the remaining cases. Because the threshold is chosen after seeing the test errors, the initial performance boost is exactly what you'd expect from selection, even if the scores are meaningless. That's the central new contribution, and there is no validation set or pre-registered threshold. The 50% containment statistic is also not calibrated: N=1000 was selected after watching the coverage curve plateau at 50% (Supplementary Figure 4), so it's not a confidence interval in any principled sense. There's also a notable internal inconsistency: the deterministic single-sample model (1.81 mm) outperforms the multi-sampling mean/median (2.15 mm), so the only benefit comes from rejection, and that rejection is in-sample. No latency measurements support the 'real-time' label, though the underlying U-Net is probably fast enough.\n\nThis is not a throwaway paper. The base measurement result is solid and the feedback concept is worth exploring. But the empirical support for the feedback is weak. I'd send it to peer review—a serious editor shouldn't desk reject it—but require an independent validation set, pre-defined thresholds or a proper calibration procedure, and a clear statement of what coverage means. If the authors can show the variance scores transfer to new data, the paper would be genuinely useful. For now, it's a promising report, not a validated system.","headline":"Base HC measurement is solid, but the variance-score feedback claim is under-validated and likely overfit to the test set.","tokens_in":7606,"tokens_out":2606,"would_cite":false,"duration_ms":28075,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Probabilistic multi-sampling gives fetal head circumference measurements confidence bounds and variance scores that guide sonographers to better scan planes in real time.","keywords":["fetal head circumference","ultrasound","probabilistic segmentation","Monte Carlo dropout","uncertainty estimation","variance score","real-time feedback","confidence bounds"],"falsifier":"Assemble a set of ultrasound images with independent ratings of view quality and ground-truth head circumference, then check whether images the variance score rejects are systematically less accurate or lower quality than images it accepts; if the score separates the two groups no better than chance, or a threshold calibrated on one scanner fails on another, the feedback loop is not carrying the claimed signal.","tokens_in":6604,"feed_emoji":"📏","tokens_out":14801,"duration_ms":139521,"temperature":0.7,"pith_summary":"This paper is trying to establish that a fetal head circumference measurement from ultrasound should come with a confidence interval and a quality score, not just a single number. It proposes a probabilistic deep network that, for one image, outputs many plausible head segmentations; fitting an ellipse to each and aggregating the results yields a head circumference plus upper and lower bounds. The spread of the sampled ellipses and masks is condensed into variance scores, and the paper shows that rejecting high-variance images improves average accuracy and that the predicted bounds contain the true measurement for about half the test images when many samples are taken. If this holds in practice, the payoff is real-time feedback that tells a sonographer when to re-scan, which could reduce the inter-operator variability that limits manual fetal ultrasound.","feed_headline":"Uncertainty score guides sonographers to better fetal head views","feed_subtitle":"A single network pass produces a head-circumference range; spread across repeated passes flags unreliable views.","key_machinery":"The machinery is a probabilistic segmentation pipeline that turns one ultrasound image into $N$ candidate head masks. Monte Carlo Dropout (running a U-Net with dropout active during inference) and a Probabilistic U-Net (a U-Net combined with a conditional variational autoencoder) act as samplers; each candidate mask is converted into an ellipse by direct least-squares fitting, and the head circumference is computed from the fitted axes using the Ramanujan II approximation. The mean or median of the $N$ circumferences becomes the reported measurement, the spread of the ellipses defines the confidence margins, and four variance scores built from those samples are thresholded to accept or reject the image. The variance-score threshold is the component that turns the sampler into an operator guidance tool.","core_discovery":"The central claim is that the disagreement among stochastic predictions of a fetal head segmentation network is informative enough to serve as a signal of measurement reliability. The authors show that Monte Carlo Dropout with dropout probability $p=0.6$ on a U-Net trained on 2,848 training images and evaluated on 540 held-out images gives $1.81 \\pm 1.65$ mm mean absolute error, matching the deterministic baseline while adding an upper and lower bound on every measurement. With $N=10$ samples, 27.8% of ground-truth head circumferences fall inside the predicted bounds; with $N=1{,}000$ samples that rises to 50.4%, and for cases outside the bounds the average distance from the true value to the margin is $1.82 \\pm 1.78$ mm. Each of the four proposed variance scores (ellipse parameter variance, ring area between the union and intersection of sampled masks, mask entropy, and softmax entropy) improves average accuracy when used to reject high-variance images, which the paper reads as evidence that higher sampled variance coincides with sub-optimal scan planes.","pith_inferences":["The paper leaves the variance-score thresholds and the containment rate as descriptive results; a natural next test is to calibrate the bounds on one dataset so that the stated confidence matches the observed rate, then apply them to a new scanner or population.","An end-to-end user study with real sonographers could test whether showing the re-scan prompt during freehand scanning actually changes the chosen plane and reduces inter-operator spread in the final measurements.","The same rejection mechanism could function as an out-of-distribution detector for abnormal anatomy, since high variance may flag images unlike the training set; the paper lists such cases as future work rather than demonstrating detection.","The variance scores are not calibrated as true uncertainty estimates, so a useful extension is to compare them against independent inter-observer disagreement on the same images to see whether high variance marks exactly the cases where human sonographers also disagree."],"forward_implications":["A deployed system could show the head circumference with its upper and lower bounds and issue a re-scan warning when the variance score crosses a threshold, pushing operators toward views the model handles well.","Rejecting high-variance images raises the average accuracy of the accepted set, so a stricter acceptance threshold trades extra scan attempts for higher measurement reliability.","Because the same dropout-equipped U-Net supplies both the measurement and the confidence information, the feedback does not require a separate uncertainty network or a slower ensemble.","Consistent use of the feedback would normalize the chosen scan plane toward the training distribution, which should make measurements more reproducible across sonographers."],"supporting_citations":[{"why":"Establishes the clinical problem of large intra- and inter-observer variability in fetal ultrasound measurements.","marker":"[12]"},{"why":"Provides the base head biometry network that the probabilistic pipeline extends.","marker":"[14]"},{"why":"Supplies Monte Carlo Dropout as an approximation to Bayesian inference, the mechanism that produces the sampled segmentations.","marker":"[5]"},{"why":"Supplies the Probabilistic U-Net, the alternative sampler compared with dropout sampling.","marker":"[7]"},{"why":"Provides the direct least-squares ellipse fitting that converts each sampled mask into ellipse parameters.","marker":"[4]"},{"why":"Gives the Ramanujan perimeter approximation used to compute head circumference from the fitted ellipse axes.","marker":"[1]"},{"why":"Supplies the U-Net architecture used as the segmentation backbone.","marker":"[10]"},{"why":"Supplies the public ultrasound dataset combined with the clinical data for training.","marker":"[15]"}],"fun_headline_variants":["Uncertainty guides sonographers to reliable fetal head scans","Real-time confidence feedback steers fetal ultrasound views","Monte Carlo dropout flags unreliable fetal head measurements","AI uncertainty helps sonographers pick optimal scan planes","Probabilistic deep learning improves fetal head measurement confidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the run-to-run variation in the network's outputs actually tracks how good the ultrasound view is, and that a cutoff for \"too variable\" learned on one set of images will keep working on new patients, machines, and anatomies.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty guides sonographers to reliable fetal head scans","Real-time confidence feedback steers fetal ultrasound views","Monte Carlo dropout flags unreliable fetal head measurements","AI uncertainty helps sonographers pick optimal scan planes","Probabilistic deep learning improves fetal head measurement confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1336,"prompt_tokens":983,"completion_tokens":353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":279}},"tokens_in":599,"tokens_out":353,"duration_ms":3918,"temperature":1.0,"reasoning_tokens":279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:39:34.050423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Assemble a set of ultrasound images with independent ratings of view quality and ground-truth head circumference, then check whether images the variance score rejects are systematically less accurate or lower quality than images it accepts; if the score separates the two groups no better than chance, or a threshold calibrated on one scanner fails on another, the feedback loop is not carrying the claimed signal.","supporting_citations":[{"cited_title":"Ultrasound Obstet Gynecol 39(3), 266–273 (mar 2012)","cited_arxiv_id":null,"evidence_quote":"Establishes the clinical problem of large intra- and inter-observer variability in fetal ultrasound measurements."},{"cited_title":"In: ICLR’16","cited_arxiv_id":null,"evidence_quote":"Supplies Monte Carlo Dropout as an approximation to Bayesian inference, the mechanism that produces the sampled segmentations."},{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the Probabilistic U-Net, the alternative sampler compared with dropout sampling."},{"cited_title":"In: 13th ICPR’96","cited_arxiv_id":null,"evidence_quote":"Provides the direct least-squares ellipse fitting that converts each sampled mask into ellipse parameters."},{"cited_title":"Journal of Mathematical Analysis and Applications 260(2), 295–306 (8 2001)","cited_arxiv_id":null,"evidence_quote":"Gives the Ramanujan perimeter approximation used to compute head circumference from the fitted ellipse axes."}],"review_version":1}