{"id":"da678848-39d1-492a-aecf-3b0ee25758b1","arxiv_id":"1908.09148","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Automatic cervical length and anterior cervical angle estimation from U-Net segmentations is presented, but the claimed 18% false-negative preterm birth rate is unsupported by the reported tables.","lead":"A Warsaw University of Technology team trained a U-Net to segment the cervix in ultrasound images and then automatically estimated cervical length and anterior cervical angle from the segmentations. The paper claims these markers reduce false-negative preterm birth predictions from 30% to 18%, but the paper's own tables do not support that figure and the prediction step uses a separate manually measured dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 18% false-negative reduction is contradicted by the paper's own Table 2: 21 false negatives among 52 actual preterm cases gives a conditional FN rate of 40.4%, not 18%.","rationale":"The key load-bearing claim is the quantitative clinical benefit: automatic marker estimation plus combined CL and ACA reduces false-negative predictions from 30% to 18%. The paper's own data fail this claim before any question of transfer to automatic markers is even considered. Table 2 gives 21 false negatives out of 52 actual preterm cases, a conditional FN rate of 40.4%, and Table 1's best recall of 74% implies 26%. The only way to obtain 18% is to use FN divided by all 114 test instances, which is incompatible with the conditional 30% baseline. Thus the headline reduction is not supported internally. The reader's weakest assumption, that the 18% result would need to transfer from manual measurements to automatic estimates, is also valid and relevant, but it is secondary: the central claim is not reproducible even on the manual-marker data. The additional pipeline mismatch in Section 3.3 further weakens the claim, as does the low ACA correlation of 0.693. The Jaccard SD inconsistency and other reporting issues reinforce low confidence. The reader's rejection verdict is appropriate, and no adjustment is needed.","tokens_in":5542,"tokens_out":7394,"duration_ms":77241,"concrete_test":"Recompute the false-negative rate from Table 2 using the same conditional definition as the 30% baseline, FN/(FN+TP). If the result is 21/52 = 40.4%, the claimed 18% reduction is not reproducible from the paper's own data. To also test whether the result transfers to the automatic pipeline, retrain the classifier on the U-Net-estimated CL and ACA markers from Section 3.2 instead of the manual measurements in Section 3.3, and report the conditional FN rate; a value near 18% would be required to substantiate the abstract claim for the proposed method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that combined CL and ACA markers decrease the false-negative ratio from 30% to 18%. The paper's own classifier results do not support this. In Table 2, 21 of the 52 actual preterm cases are predicted as control, giving a conditional false-negative rate FN/(FN+TP) = 21/52 = 40.4%, not 18%. The only way to obtain approximately 18% from Table 2 is to divide by all 114 test instances (21/114 = 18.4%, 16/114 = 14.0%), but the 30% baseline is defined conditionally on spontaneous preterm deliveries, so switching denominators invalidates the comparison. Table 1 also fails to support the claim: the best Naive Bayes I+II result has recall 74%, implying a 26% conditional FN rate, while the first-trimester recall of 69% implies 31%. No reported configuration yields the headline 18%. Additionally, Section 3.3 states that the classifiers were trained on 380 precomputed manual CL and ACA measurements, not on the U-Net-estimated markers evaluated in Section 3.2; the automatic ACA estimates have correlation 0.693 and RMSE 16.22 degrees, so even a valid 18% result on manual markers would not validate the end-to-end automatic pipeline described in the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic pipeline for estimating two ultrasound markers of preterm birth risk, cervical length (CL) and anterior cervical angle (ACA), from transvaginal ultrasound images. The pipeline first segments the cervix with a U-Net, then estimates CL via a centerline algorithm and ACA via recursive centroid splitting, and finally uses these markers in binary classifiers (SVM, KNN, Naive Bayes, decision trees) to distinguish preterm from control pregnancies. On a separate dataset of 380 precomputed manual measurements, the authors report that combining CL and ACA decreases the false-negative ratio from 30% to 18%. The abstract presents this as the main result and frames the method as an objective alternative to subjective visual assessment by gynecologists.","tokens_in":5773,"tokens_out":6541,"duration_ms":65536,"significance":"If the end-to-end pipeline were validated, automatic and objective extraction of CL and ACA could have practical value for preterm-birth screening, and the paper would address an important clinical problem. The authors are transparent about the weak correlation of their automatic ACA estimates with ground truth (correlation 0.693, RMSE 16.22 degrees), and they report training details for the U-Net in reasonable detail. However, the central claimed clinical outcome is contradicted by the paper's own tables, and the classifier evaluation is not performed on the automatic marker estimates that the paper advertises. As submitted, the manuscript does not support the abstract's main claim and requires substantial additional experiments and corrections before it could be considered for publication.","major_comments":[{"comment":"The abstract's central claim that combined CL and ACA markers decrease the false-negative ratio from 30% to 18% is not supported by the results in Table 2. The confusion matrix shows 21 false negatives among 52 actual preterm cases, giving a conditional false-negative rate of 21/52 = 40.4%, not 18%. The value 18% appears only when 21 is divided by all 114 test instances (21/114 = 18.4%), but the baseline of 30% is defined conditionally on spontaneous preterm deliveries, so changing the denominator invalidates the comparison. Table 1 also contradicts the claim: the best Naive Bayes classifier with first- and second-trimester features has recall 74%, implying 26% false negatives, and the first-trimester Naive Bayes classifier has recall 69%, implying 31% false negatives. Therefore the text in Section 3.3 stating that \"for the first trimester, we obtained 18% of false negatives\" is unsupported by the paper's own data.","section":"Abstract and Section 3.3 (Table 2)"},{"comment":"The classifiers are trained and evaluated on a different dataset of 380 balanced numerical samples with precomputed CL and ACA measurements, not on marker values estimated automatically by the U-Net pipeline described in Sections 3.1 and 3.2. Because the automatic ACA estimates have a correlation of only 0.693 with ground truth and an RMSE of 16.22 degrees, classifier performance measured on manual measurements cannot be assumed to transfer to the automatic estimates. The abstract claims that the proposed automatic pipeline allows the false-negative ratio to be reduced to 18%, but no experiment in the paper evaluates the complete pipeline from ultrasound image to classification. An end-to-end evaluation on automatically estimated markers is missing and is load-bearing for the paper's main claim.","section":"Section 3.3 (first paragraph) and Abstract"},{"comment":"The reported segmentation performance in Section 3.1 is internally inconsistent. The average Jaccard Index is reported as 0.91 with minimum 0.89, maximum 0.92, and standard deviation 0.1. For any distribution taking values in the interval [0.89, 0.92], the standard deviation cannot exceed 0.015 (half the range width), so a reported standard deviation of 0.1 is mathematically impossible. This indicates an error in either the computation or the reporting of the segmentation statistics and makes the claimed segmentation quality unreliable as stated.","section":"Section 3.1 (Jaccard Index results)"},{"comment":"The comparison between the reported results and the baseline from reference [10] is not a fair or meaningful comparison. The paper contrasts its detection rate of 74% with the 54.8% detection rate reported in [10], but reference [10] reports detection at a fixed 10% false-positive rate, whereas the present study uses a balanced dataset and does not report performance at a matched false-positive rate. The authors acknowledge a higher false-negative ratio than [10] but do not provide an operating-point analysis such as ROC curves at comparable false-positive rates. Without this, the claimed superiority over prior screening methods is not established.","section":"Section 3.3 (Table 1 and comparison to [10])"}],"minor_comments":[{"comment":"The RMSE of 110.88 for cervical length estimation is reported without units. If the values are in millimeters, such an error would be clinically implausible; if they are in pixels or some other unit, the unit should be stated explicitly.","section":"Section 3.2 (CL estimation)"},{"comment":"The text states \"We start training a U-Net model\" and contains fragments such as \"to overcome fact that our ultrasound dataset\"; these should be corrected for grammatical clarity.","section":"Section 3.1 (Experimental settings)"},{"comment":"The performance metrics in Table 1 are not accompanied by confidence intervals or standard deviations, despite the paper mentioning 5-fold cross-validation. Reporting the variability across folds would help assess whether the differences between classifiers are meaningful.","section":"Section 3.3 (Table 1)"},{"comment":"The U-Net architecture diagram is difficult to interpret because the feature-map dimensions are not clearly labeled along both axes and the caption is too brief to explain the visual encoding.","section":"Figure 2"}],"recommendation":"reject","confidential_remarks":"This manuscript reports a promising direction, but the central clinical claim is contradicted by the paper's own confusion matrix, and the classifier evaluation is performed on a separate manual-measurement dataset rather than on the automatic pipeline's outputs. There is also an impossible reported standard deviation. A methods-focused paper on automatic CL and ACA extraction might be viable after substantial revision and an end-to-end evaluation, but the current submission does not meet the journal's standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know that the paper's headline result—reducing preterm-birth false negatives from 30% to 18%—is not supported by its own tables. That is the main thing.\n\nThe idea of automatically estimating the anterior cervical angle (ACA) from a segmentation mask via recursive centroid splitting is new as far as I can tell, and the workflow of inpainting annotation marks before training U-Net is a sensible practical step. The segmentation results are plausible in isolation, though the reported Jaccard SD of 0.1 is mathematically impossible given a min of 0.89 and a max of 0.92.\n\nBut the inconsistencies are load-bearing. Table 2's confusion matrix shows 21 false negatives out of 52 actual preterm cases (40.4%), not 18%. The 18% figure appears in the abstract and again in Section 3.3, but no reported configuration yields it. The best Naive Bayes result in Table 1 has recall 74%, implying a 26% false-negative rate. And Section 3.3 says the classifiers were trained on 380 precomputed manual measurements, while Table 2 sums to 114 samples—so the matrix and the dataset size disagree. These are not minor oversights; they strike at the central claim.\n\nEven if the 18% result were valid, it comes from a separate dataset with manually precomputed markers, not from the U-Net outputs. The automatic ACA estimates have a 0.693 correlation and an RMSE of 16.22 degrees, so classification performance on manual measurements cannot be assumed to transfer to the automatic pipeline. The end-to-end system described in the abstract is never evaluated as such.\n\nA reader interested in the segmentation or marker-estimation pieces might find something worth a look, but the paper as a whole cannot sustain its central claim. It is not ready for peer review in its current form; the authors would need to reconcile the tables, rerun the evaluation on the actual pipeline, and report a consistent set of numbers. My recommendation is to reject it. If they fix these issues and resubmit, it might become worth revisiting.","headline":"The paper's central false-negative claim crumbles under its own confusion matrix, and the evaluation is not end-to-end, so the stated clinical benefit does not hold up.","tokens_in":6369,"tokens_out":2709,"would_cite":false,"duration_ms":26858,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A U-Net pipeline measuring cervical length and anterior cervical angle predicts preterm birth with an 18% false-negative rate, down from 30% for manual screening.","keywords":["preterm birth prediction","cervical length","anterior cervical angle","U-Net segmentation","transvaginal ultrasound","deep learning","naive Bayes classifier"],"falsifier":"Apply the trained naive Bayes classifier to the CL and ACA values produced automatically by the U-Net pipeline for the 52 preterm and 62 control pregnancies, and compare the confusion matrix with the one obtained from the 380 precomputed samples. A false-negative count of 21 out of 52 (40%) would show that the 18% figure does not transfer; alternatively, a paired test on both manual and automatic markers would settle whether segmentation accuracy is the limiting factor.","tokens_in":5351,"feed_emoji":"🤰","tokens_out":11729,"duration_ms":94028,"temperature":0.7,"pith_summary":"Preterm birth is the most common cause of neonatal death, and current ultrasound screening misses roughly 30% of spontaneous preterm deliveries because it depends on visual inspection. This paper tries to replace that subjectivity with an automatic pipeline: a U-Net segments the cervix from transvaginal ultrasound images, geometric algorithms measure cervical length (CL) and anterior cervical angle (ACA) from the masks, and a naive Bayes classifier turns the two numbers into a preterm/control prediction. The authors report a mean segmentation Jaccard index of 0.91, CL estimates that correlate at 0.94 with expert measurement, ACA estimates that correlate at 0.693, and a classifier that lowers the false-negative ratio to 18% at a 14% false-positive rate when CL and ACA are combined. If correct, clinicians could get objective, repeatable risk scores without relying on a gynaecologist's eye. The classifier result is presented on a separate 380-sample dataset of precomputed markers, so the transfer from expert-measured to automatically measured markers is the key step to verify.","feed_headline":"Automatic ultrasound markers cut missed preterm births to 18 percent","feed_subtitle":"A U-Net measures cervical length and angle from ultrasound images, offering objective preterm risk screening.","key_machinery":"The central machinery is the U-Net, an encoder-decoder convolutional network for biomedical image segmentation, which produces a binary mask of the cervix from each preprocessed ultrasound image. Cervical length is then estimated by a centerline algorithm that builds a Voronoi skeleton of the mask and smooths the selected centerline; anterior cervical angle is estimated by a three-iteration recursive centroid split that divides the mask and measures the angle between the anterior wall and the line through the last two centroids. A naive Bayes classifier maps the resulting (CL, ACA) pairs to preterm/control labels. This chain converts subjective visual features into numerical markers and then into a risk decision.","core_discovery":"The central claim is that a complete deep-learning pipeline can extract the two ultrasound markers used in preterm birth screening—cervical length and anterior cervical angle—from raw transvaginal ultrasound images, and that combining these markers predicts spontaneous preterm birth better than current manual screening. On the segmentation task the U-Net achieves an average Jaccard index of 0.91. The cervical-length estimate has correlation 0.94 with ground truth (RMSE 110.88), and the anterior-cervical-angle estimate has correlation 0.693 (RMSE 16.22). Using first- and second-trimester (CL, ACA) pairs, a naive Bayes classifier reaches 77.5% accuracy, 85% precision, 74% recall, and 0.78 AUC, which the paper reports as 18% false negatives versus about 30% for manual screening and 14% false positives. The intended upshot is an objective, repeatable alternative to visual assessment, with the same segmentation masks available for other biophysical markers such as cervical shape and tissue density.","pith_inferences":["The 18% false-negative figure comes from expert-measured markers on a separate 380-sample dataset; if the automatic ACA estimates (correlation 0.693) feed the classifier, the real false-negative rate could be substantially higher, so an end-to-end evaluation is the decisive test the paper leaves open.","The recursive centroid-split ACA algorithm is sensitive to segmentation boundary noise; evaluating it on masks with artificially degraded Jaccard indices would show how much of the prediction quality depends on the U-Net rather than on the classifier.","Because the classifier was trained on a balanced 50:50 preterm/control set, the reported 14% false-positive and 18% false-negative rates do not directly reflect clinical prevalence; recalibrating to the observed ~20% preterm rate would change both rates."],"forward_implications":["Ultrasound screening for preterm risk could become a fully automatic pipeline: image in, risk score out, removing the subjective step that currently misses about 30% of spontaneous preterm deliveries.","Because the segmentation masks are reusable geometry, the same network can support additional markers such as cervical shape or tissue density without retraining the segmentation stage.","Adding anterior cervical angle to cervical length gives a concrete way to improve first-trimester prediction, the period in which current screening is least effective.","A detection rate near 74% at a 10% false-positive rate would outperform the 54.8% detection rate of the standard maternal-factor-and-placental-perfusion model, making earlier and more reliable risk stratification possible."],"supporting_citations":[{"why":"Supplies the U-Net encoder-decoder architecture used to segment cervical shapes from ultrasound images.","marker":"[11]"},{"why":"Introduces anterior cervical angle as a biophysical marker for spontaneous preterm birth, the feature the paper combines with cervical length.","marker":"[12]"},{"why":"Provides the centerline algorithm applied to binary masks for estimating cervical length.","marker":"[13]"},{"why":"Reports a 54.8% detection rate at a 10% false-positive rate, the baseline the paper compares its 74% recall against.","marker":"[10]"},{"why":"Documents the global burden of preterm birth that motivates the prediction problem.","marker":"[1]"},{"why":"Establishes cervical length and obstetric history as the standard screening model that the paper's CL+ACA combination extends.","marker":"[3]"}],"fun_headline_variants":["Deep learning cuts missed preterm births to 18 percent","U-Net extracts cervical markers to slash preterm false negatives","Objective preterm birth risk from deep learning on ultrasound scans","Preterm birth screening improved by automated ultrasound segmentation","From scan to marker: AI boosts preterm birth prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the classifier's 18% false-negative rate, obtained from a separate 380-sample dataset of expert-measured cervical lengths and angles, will also hold for the automatically segmented and estimated markers, even though the automatic angle estimates correlate only 0.693 with expert measurement.","fun_headline_variants_meta":{"raw":{"variants":["Deep learning cuts missed preterm births to 18 percent","U-Net extracts cervical markers to slash preterm false negatives","Objective preterm birth risk from deep learning on ultrasound scans","Preterm birth screening improved by automated ultrasound segmentation","From scan to marker: AI boosts preterm birth prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2994,"prompt_tokens":974,"completion_tokens":2020,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1944}},"tokens_in":590,"tokens_out":2020,"duration_ms":16253,"temperature":1.0,"reasoning_tokens":1944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:20:08.023508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the trained naive Bayes classifier to the CL and ACA values produced automatically by the U-Net pipeline for the 52 preterm and 62 control pregnancies, and compare the confusion matrix with the one obtained from the 380 precomputed samples. A false-negative count of 21 out of 52 (40%) would show that the 18% figure does not transfer; alternatively, a paired test on both manual and automatic markers would settle whether segmentation accuracy is the limiting factor.","supporting_citations":[{"cited_title":"et al.: U-net: Convolutional networks for biomedical image segmen- tation","cited_arxiv_id":null,"evidence_quote":"Supplies the U-Net encoder-decoder architecture used to segment cervical shapes from ultrasound images."},{"cited_title":"Ultrasound in Obstetrics and Gynecology 46, 377—378 (2015)","cited_arxiv_id":null,"evidence_quote":"Introduces anterior cervical angle as a biophysical marker for spontaneous preterm birth, the feature the paper combines with cervical length."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the centerline algorithm applied to binary masks for estimating cervical length."},{"cited_title":"et al.: Prediction of spontaneous preterm delivery from maternal factors, obstetric history and placental perfusion and function at 11-13 weeks","cited_arxiv_id":null,"evidence_quote":"Reports a 54.8% detection rate at a 10% false-positive rate, the baseline the paper compares its 74% recall against."},{"cited_title":"Born Too Soon: The Global Action Report on Preterm Birth","cited_arxiv_id":null,"evidence_quote":"Documents the global burden of preterm birth that motivates the prediction problem."},{"cited_title":"et al.: Cervical length and obstetric history predict spontaneous preterm birth: development and validation of a model to provide individualized risk assess- ment","cited_arxiv_id":null,"evidence_quote":"Establishes cervical length and obstetric history as the standard screening model that the paper's CL+ACA combination extends."}],"review_version":1}