{"id":"634b0cb7-b155-4101-9fcb-a1dcae5df91c","arxiv_id":"2506.08623","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-branch EfficientNet ensemble trained with LDAM-Focal loss reaches 85% accuracy over 16 fetal ultrasound classes, but the abstract overstates per-class results and the dataset is not public.","lead":"This paper trains a two-branch deep learning ensemble to sort second-trimester fetal ultrasound images into 16 anatomical classes, reporting 85% overall accuracy on a new 5,298 image dataset. A reader should check it because the abstract's per-class performance claims do not match the paper's own result table, and no code or data is released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out test set is described; with ~60 tuning experiments, the reported accuracy may reflect test-set overfitting. Table 3 also contradicts the abstract's 75% >0.85 claim.","rationale":"The paper's central quantitative claims—85% accuracy, 0.86 F1, 55% on the hardest class, and the 90%/75% aggregate statements—all depend on the reported evaluation being an unbiased estimate of generalization. The manuscript never describes a train/validation/test split, despite reporting roughly sixty tuning experiments and selecting the best configuration. This is the weakest link because a clean split is a necessary condition for any of the reported numbers to be meaningful, and it is entirely absent. I agree with the reader's assessment that this makes the paper conditional rather than accepted. I also note two internal inconsistencies (Table 3 vs the abstract, and the impossible expert-overlap counts in §4) that strengthen the concern, but the missing split is the more fundamental issue. A concrete remedy would be to disclose the split and re-evaluate the selected model once on a truly held-out test set.","tokens_in":7842,"tokens_out":6259,"duration_ms":72476,"concrete_test":"Request from the authors the exact per-class train/validation/test split and the evaluation protocol; if the test set was used during the sixty-experiment model selection, re-run the final EfficientNet-B0+B6 LDAM-Focal configuration once on a truly held-out test set and report accuracy/F1. If the performance drops materially below 85%/0.86, the central claim is overturned.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that approximately sixty experiments were run and the best configuration was selected, with Table 2 listing top-10 results. Nowhere in §2 (Dataset Description, Metrics) or §3 is the division of the 5,298 images into training, validation, and test sets described. No confidence intervals are provided. If the same labelled images were used both to choose among sixty configurations and to produce the reported metrics, the headline 85% accuracy, 0.86 F1, and 55% Umbilical Cord (AAW) accuracy are optimistically biased estimates of generalization, and the loss-function comparisons in Table 2 are not a valid model-selection procedure. This is the most load-bearing issue because it affects every quantitative claim in the paper, including the 90%/75% aggregate statements. The concern is reinforced by internal arithmetic problems: Table 3 yields 14/16 classes above 0.75 (87.5%, not 90%) and only 7/16 above 0.85 (43.75%, not 75%); the expert-agreement experiment in §4 reports 492 overlapping positive annotations when Expert B marked only 455 images, which is impossible for an intersection. These discrepancies suggest the reported numbers have not been carefully cross-checked, making the missing evaluation protocol especially consequential.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a dual-branch ensemble (EfficientNet-B0 as a 'shallow' branch and EfficientNet-B6 as a 'detailed' branch, combined with LDAM-Focal loss) for classifying 16 fetal anatomical structures in second-trimester ultrasound images. The authors describe a 5,298-image dataset annotated by three clinicians and reconciled with the Dawid-Skene method, on-the-fly augmentations, and a comparison of five loss functions. The headline results are 85% overall accuracy, 0.86 F1-score, 55% accuracy on the hardest class (Umbilical Cord Anterior Abdominal Wall), and a claimed correspondence between this 55% and a measured 54% human expert agreement. They also claim in the abstract that 90% of organs achieve accuracy above 0.75 and 75% above 0.85.","tokens_in":8142,"tokens_out":4129,"duration_ms":48171,"significance":"If supported by a rigorous evaluation, this would be a useful contribution to fetal ultrasound classification: the 16-class scope is broader than many prior studies, the use of routinely acquired noisy clinical images is realistic, and the comparison of imbalance-aware losses is practically relevant. The dataset and Dawid-Skene annotation procedure are assets. However, the current manuscript contains major reporting inconsistencies and an underspecified evaluation protocol that, as it stands, prevent the headline quantitative claims from being accepted at face value.","major_comments":[{"comment":"The paper never describes how the 5,298 images are divided into training, validation, and test sets. Section 3 states that 'approximately sixty experiments' were run and the best configuration was selected, with Table 2 listing the top ten results. If the reported metrics were computed on data that also influenced the selection of the architecture, loss, and hyperparameters, then the 85% accuracy, 0.86 F1, and 55% class accuracy are optimistically biased estimates of generalization. A precise description of the split (including whether it is image-level or patient-level) and confirmation that the test set was used only once are required for the central claim.","section":"§2.1, §3"},{"comment":"The abstract and discussion claim that 90% of organs have accuracy above 0.75 and 75% have accuracy above 0.85, but Table 3 does not support these numbers. Counting from Table 3, 14 of 16 classes (87.5%) exceed 0.75 (exceptions: 55% and 70%), and only 8 of 16 classes (50%) exceed 0.85. The authors should either recompute the aggregate percentages or correct the text.","section":"Abstract, §4"},{"comment":"The expert-agreement experiment is internally impossible as reported: Expert A identified 880 images as positive, Expert B marked 455 images, yet the paper states that 492 annotations overlapped. Since an intersection cannot exceed the smaller set (455), the reported overlap is arithmetically inconsistent. The 54% agreement figure needs to be recomputed from the correct overlap or the description of the annotations must be revised.","section":"§4"},{"comment":"No confidence intervals, standard deviations, or repeated-run variability are reported for any metric. Given the class imbalance and the small size of the hardest class (149 images for Umbilical Cord Anterior Abdominal Wall), the 55% accuracy is a single point estimate whose uncertainty should be quantified. Reporting bootstrap or repeated-seed intervals would substantially strengthen the evaluation.","section":"§2.4, Table 2"}],"minor_comments":[{"comment":"Class labels are inconsistent with Table 1: 'Shoulder bone' appears in Table 3 but not in Table 1, which lists 'Humerus'; 'Nasal triangle' appears instead of 'Nasolabial triangle'; and 'Placenta (Umbilical cord)' differs from 'Umbilical cord (placenta)' in Table 1.","section":"Table 3"},{"comment":"In the first bullet of Section 2.2, the sentence 'In some cases, scaling factors are applied rather than fixed dimensional rescaling.' is incomplete and should be removed or expanded.","section":"§2.2"},{"comment":"The Focal Loss reference is attributed to 'Ross and Dollár, 2017'; the correct author list is Lin, Goyal, Girshick, He, and Dollár. This should be corrected.","section":"References"},{"comment":"The paper reports 'approximately sixty experiments' but does not specify hyperparameter ranges, augmentation parameter ranges, or random seeds, making the results difficult to reproduce independently.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claims are currently not self-consistent, and the missing evaluation protocol is the most serious issue. I would not reject outright because the errors are fixable in principle: the authors can supply a proper held-out test description, recompute the aggregate percentages, correct the expert-overlap arithmetic, and re-run or re-report uncertainties. However, if the reported numbers were produced without a separation between model selection and evaluation, the quantitative conclusions would need to be regenerated under a valid protocol. Given the journal context, dataset and code availability would also materially help the reviewers verify the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2506.08623. The genuinely new thing is the dataset: 5,298 images, 16 fetal ultrasound classes, with Dawid-Skene consensus labels from three experts. That's a step beyond the usual six-class benchmarks, and the class distribution reflects real acquisition conditions. The authors also deserve credit for framing model performance against inter-expert agreement on the hardest class—that's the right way to think about annotation noise.\n\nThe architecture itself is not new: a shallow and a deep EfficientNet branch with concatenated features is a standard multi-scale ensemble, and the \"biologically inspired\" wording adds no technical content. That's a minor complaint.\n\nThe bigger problems are internal. The abstract says 75% of organs have accuracy above 0.85; Table 3 shows 7 out of 16 classes (43.75%) above that threshold. Even the 90% above 0.75 claim is off (14/16, 87.5%). And in the expert-agreement experiment, Expert B marked 455 images positive but the overlap is reported as 492—mathematically impossible. Those aren't style issues; they suggest the numbers haven't been thoroughly cross-checked.\n\nThe load-bearing flaw is the missing evaluation protocol. No train/val/test split is described. Roughly sixty configurations were tried and the best selected by looking at these metrics. Without a clear statement that the final numbers come from an untouched test set, the headline accuracy and the loss-function comparisons are optimistic estimates. The absence of confidence intervals makes it worse.\n\nI don't think the paper is fraudulent; the approach is plausible and the dataset likely exists. But as written, the quantitative claims are unreliable. The good news is these problems are fixable: describe the split, correct the arithmetic, release code or at least the data, and the paper could be a useful dataset contribution.\n\nWho's this for? Anyone working on fetal ultrasound CAD, and instructors who want a clean example of why evaluation hygiene matters. I'd send it to peer review because the dataset and the question deserve referee time, but I'd expect major revision before acceptance.","headline":"The 16-class dataset is a step forward, but the paper's reported numbers don't survive contact with its own tables, and the missing split protocol undermines every quantitative claim.","tokens_in":8646,"tokens_out":2370,"would_cite":false,"duration_ms":27655,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A biologically inspired dual-branch ensemble classifies 16 second-trimester fetal ultrasound structures at 85% accuracy, and on the hardest class it matches the 54% agreement rate of expert clinicians.","keywords":["fetal ultrasound","image classification","EfficientNet","ensemble learning","class imbalance","LDAM-Focal loss","Dawid-Skene","biologically inspired networks"],"falsifier":"A reader who counts the per-class accuracies in Table 3 finds 7 of 16 classes above 85% and 14 above 75%, not the 75% and 90% the abstract claims; an independent held-out evaluation on data never used for model selection would settle whether the overall 85% accuracy holds.","tokens_in":7656,"feed_emoji":"🩺","tokens_out":8189,"duration_ms":91145,"temperature":0.7,"pith_summary":"This paper sets out to show that a lightweight, biologically inspired ensemble can classify 16 fetal anatomical structures from real second-trimester ultrasound scans, a much broader task than prior work aimed at a few standard views. The authors built a dataset of 5,298 clinical images with consensus labels from three clinicians, and combined a coarse-feature branch (EfficientNet-B0) with a fine-feature branch (EfficientNet-B6) under an imbalance-aware LDAM-Focal loss. Their best configuration reaches 85% overall accuracy and a 0.86 F1-score, with 12 of 16 classes above 80% accuracy. On the most ambiguous class, the umbilical cord at the anterior abdominal wall, the model's 55% accuracy matches the 54% agreement rate between two expert annotators, which the authors take as evidence that remaining error is bounded by human-level label ambiguity rather than model capacity.","feed_headline":"Ultrasound AI reads 16 fetal structures at 85% accuracy","feed_subtitle":"Dual-branch network also matches human agreement on the hardest class, umbilical cord attachment.","key_machinery":"The load-bearing mechanism is the two-branch ensemble: the same input image is resized to a low-resolution version for a shallow EfficientNet-B0 branch and to a high-resolution version for a detailed EfficientNet-B6 branch, and the global-average-pooled feature vectors of both branches are concatenated before classification. This coarse-to-fine stacking mirrors the hierarchical organization of biological vision, while the LDAM-Focal loss combines a label-distribution-aware margin with focal weighting to push performance on underrepresented classes. The ground truth itself is produced by reconciling three clinicians' annotations with the Dawid-Skene algorithm, and on-the-fly augmentation (gamma correction, random crops, flips, color jitter, blur, translation) is used to simulate real-world acquisition variability.","core_discovery":"The paper's central claim is that a dual-branch ensemble which first captures coarse, low-resolution context and then refines it with high-resolution detail can simultaneously distinguish 16 second-trimester fetal structures in noisy, routinely acquired ultrasound images. Trained with LDAM-Focal loss, the EfficientNet-B0 plus EfficientNet-B6 ensemble achieves 85% overall accuracy and 0.86 F1-score, with per-class accuracy above 80% for 12 of 16 anatomical categories. On the hardest class, umbilical cord attachment at the anterior abdominal wall, the model reaches 55% accuracy, closely matching the 54% inter-expert agreement measured in a supplementary labeling experiment; the authors interpret this as reaching the practical accuracy limit set by annotation ambiguity rather than by model capacity.","pith_inferences":["A consequence the authors leave implicit is that if the hardest class truly has only about 54% human agreement, then per-image accuracy on that class is not a well-defined target; the 55% result is perhaps better interpreted as agreement with noisy labels than as classification error.","The dual-branch design suggests a natural extension: instead of only concatenating pooled features, one could add an anatomical-prior layer that models co-occurrence (for example, umbilical cord near the anterior abdominal wall) to resolve classes that often appear as secondary structures.","Because the paper does not report the train/validation/test split, a straightforward next experiment is to freeze the final configuration and evaluate on an independent multi-site test set; the 85% figure would be materially stronger after such confirmation."],"forward_implications":["A lightweight two-branch model can cover 16 fetal anatomical classes simultaneously, not just a handful, at 85% overall accuracy and 0.86 F1-score.","On the umbilical-cord anterior-abdominal-wall class, the model's 55% accuracy tracks the 54% inter-expert agreement, so further improvement on that class likely requires cleaner label definitions or additional imaging information, not larger models.","The combination of LDAM and focal loss is the key to balanced performance on underrepresented classes, with per-class accuracy exceeding 80% for 12 of 16 structures.","This level of multi-class coverage makes the approach a candidate for real-time, low-cost decision support in second-trimester screening, especially in settings where sonographer expertise is scarce."],"supporting_citations":[{"why":"Supplies the EM-based algorithm that reconciles the three clinicians' annotations into consensus labels used as ground truth.","marker":"Dawid and Skene, 1979"},{"why":"Provides the public fetal ultrasound dataset and standard-plane framing that motivates the 16-class task.","marker":"Burgos-Artizzu et al., 2020"},{"why":"Defines the EfficientNet backbone family used for both the shallow and detailed branches.","marker":"Tan and Le, 2019"},{"why":"Introduces the label-distribution-aware margin (LDAM) loss that the best configuration combines with focal loss.","marker":"Cao et al., 2019"},{"why":"Introduces focal loss for class imbalance, which is the other half of the selected LDAM-Focal loss.","marker":"Ross and Dollár, 2017"},{"why":"Supplies the combined LDAM-Focal loss formulation used in the final ensemble.","marker":"Sadi et al., 2022"},{"why":"Represents prior standard-plane detection work with few classes that this 16-class system extends.","marker":"Baumgartner et al., 2017"}],"fun_headline_variants":["Ultrasound AI identifies 16 fetal structures with 85% accuracy","Dual-branch network matches human agreement on hardest fetal class","Biologically inspired ensemble reads 16 fetal anatomy classes","Lightweight AI ensemble classifies 16 fetal ultrasound structures","Fetal ultrasound AI hits 85% accuracy and expert-level hardest class"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies are treated as reliable estimates of performance even though the paper does not describe the train/validation/test split, gives no confidence intervals, and selected the final configuration after roughly sixty experiments on the same dataset.","fun_headline_variants_meta":{"raw":{"variants":["Ultrasound AI identifies 16 fetal structures with 85% accuracy","Dual-branch network matches human agreement on hardest fetal class","Biologically inspired ensemble reads 16 fetal anatomy classes","Lightweight AI ensemble classifies 16 fetal ultrasound structures","Fetal ultrasound AI hits 85% accuracy and expert-level hardest class"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001593,"raw_usage":{"total_tokens":6345,"prompt_tokens":934,"completion_tokens":5411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":5324}},"tokens_in":550,"tokens_out":5411,"duration_ms":38168,"temperature":1.0,"reasoning_tokens":5324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:05:17.376009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader who counts the per-class accuracies in Table 3 finds 7 of 16 classes above 85% and 14 above 75%, not the 75% and 90% the abstract claims; an independent held-out evaluation on data never used for model selection would settle whether the overall 85% accuracy holds.","supporting_citations":[{"cited_title":", author Skene, A.M","cited_arxiv_id":null,"evidence_quote":"Supplies the EM-based algorithm that reconciles the three clinicians' annotations into consensus labels used as ground truth."},{"cited_title":", author Coronado-Guti \\'e rrez, D","cited_arxiv_id":null,"evidence_quote":"Provides the public fetal ultrasound dataset and standard-plane framing that motivates the 16-class task."},{"cited_title":", author Wei, C","cited_arxiv_id":null,"evidence_quote":"Introduces the label-distribution-aware margin (LDAM) loss that the best configuration combines with focal loss."},{"cited_title":", author Doll \\'a r, G","cited_arxiv_id":null,"evidence_quote":"Introduces focal loss for class imbalance, which is the other half of the selected LDAM-Focal loss."},{"cited_title":"LMFLOSS: A Hybrid Loss For Imbalanced Medical Image Classification","cited_arxiv_id":"2212.12741","evidence_quote":"Supplies the combined LDAM-Focal loss formulation used in the final ensemble."},{"cited_title":", author Kamnitsas, K","cited_arxiv_id":null,"evidence_quote":"Represents prior standard-plane detection work with few classes that this 16-class system extends."}],"review_version":1}