{"id":"4fa296a8-e910-4766-89a1-315d87a91e20","arxiv_id":"2608.04766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FUSEP is a new multi-center public benchmark with 4,017 early-pregnancy ultrasound images, 45,820 box annotations of 14 structures, and detection baselines across four learning paradigms.","lead":"This paper releases FUSEP, a benchmark dataset of 4,017 early-pregnancy fetal ultrasound images from three hospitals with 45,820 box-level annotations covering 14 anatomical structures in the CRL and NT views. It also runs detection baselines in supervised, semi-supervised, domain adaptation, and source-free settings, giving researchers a first public testbed for this clinical stage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's ground truth is only as solid as its unmeasured labels: Sec. 3.2 asserts an exclusion-based quality gate with no IAA metrics or counts, and the four-vertex box format leaves the mAP-IoU definition unspecified, so all headline counts and baseline tables inherit this uncertainty.","rationale":"The reader's weakest-assumption analysis correctly identifies the load-bearing premise: the benchmark's ground truth is the expert annotation, and its quality is asserted without measurement. I agree with the conditional verdict. The paper's headline arithmetic is internally consistent (4,017 images; 45,820 boxes; 1,816 CRL/2,201 NT), the benchmark protocols are standard in form, and the promised GitHub artifacts are a genuine reproducibility asset, so this is not a rejection. However, the manuscript itself supplies only the sentence in Sec. 3.2 about excluding inconsistent annotations, with no IAA statistic, no exclusion count, and no per-class agreement. Because the baseline tables report large performance gaps on small structures, unmeasured label noise could masquerade as detection difficulty. The four-vertex annotation description adds a second protocol ambiguity: if the stored boxes are quadrilaterals, the mAP-50 computation needs an explicit polygon-to-axis-aligned conversion rule, and none is stated. A re-annotation study on a stratified subset, together with an evaluation-script audit, would settle both points. Until then, the benchmark is a promising but conditionally acceptable resource: its utility depends on disclosures that are currently absent, not on any demonstrated error in the released data.","tokens_in":29975,"tokens_out":12063,"duration_ms":144046,"concrete_test":"Select a stratified random subset (e.g., 200 CRL + 200 NT images), have two independent sonographers blind to the released labels re-annotate all 14 structures, and compute per-structure Cohen's kappa and IoU agreement against the published boxes; simultaneously, run Faster R-CNN on the official release with both raw-quadrilateral and axis-aligned-converted IoU and compare mAP. If per-structure agreement falls below 0.8 kappa or 0.5 IoU for any headlined structure, or if the two IoU conventions move mAP by more than the 0.2-point margins that separate the top methods in Table 3, the headline '45,820 box-level expert annotations' and the benchmark rankings are not a reliable reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 makes the benchmark's central validity claim in one unquantified sentence: 'Any inconsistent annotations were identified and excluded during the preprocessing stage to ensure high-quality labeling.' No inter-annotator agreement metric, no per-structure disagreement, and no exclusion count are given, and the two sonographers' independent labels are not released. The 45,820 boxes are the sole ground truth for every mAP in Tables 3-6. If the exclusion process was aggressive, the published cohort is a filtered subset and the reported multi-center difficulty is not representative; if disagreements concentrated on small structures (NB, NTAPS, MDS, G), the low APs in Tables 3-6 conflate label noise with detection difficulty. A second, related unstated assumption is the annotation format: Sec. 3.2 describes boxes 'with four vertices per structure,' but the baselines (MMDetection) output axis-aligned boxes, and the paper never states whether IoU is computed on the raw quadrilaterals or after conversion to axis-aligned rectangles. This ambiguity affects the definition of mAP-50 for every row of Tables 3-6. Both issues are load-bearing because the benchmark's claim to provide 'standardized detection baselines' requires a stable, well-defined ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FUSEP is a new multi-center benchmark for early-pregnancy fetal ultrasound screening, consisting of 4,017 CRL and NT ultrasound images from three hospitals with 45,820 box-level expert annotations of 14 anatomical structures. The paper reports fully supervised, semi-supervised (5% and 10% labels), unsupervised domain adaptation, and source-free domain adaptation object-detection baselines, implemented in MMDetection, and claims to be the first publicly available dataset and benchmark for this specific niche. The headline arithmetic is internally consistent, and the authors provide code, baseline weights, and metadata, plus an appendix with per-class counts and area ratios.","tokens_in":30034,"tokens_out":4274,"duration_ms":48497,"significance":"If the ground-truth quality and evaluation conventions are made explicit and verifiable, FUSEP is a potentially valuable resource for the medical image analysis community. It addresses a genuinely underrepresented setting (early pregnancy, 11–14 weeks) with two guideline-recommended views, multi-center acquisition, and two devices, and it ships reproducible artifacts: training scripts, environment configurations, baseline weights, and structured JSON metadata. The breadth of protocols (supervised, SSL, UDA, SFDA) is a useful service to the community. The explicit failure-mode analysis around extreme scale variation, class imbalance, and feature sparsity is also a strength. The current manuscript, however, does not yet substantiate the label-quality assumptions on which all benchmark numbers rest, and it leaves the exact definition of the detection metric underspecified.","major_comments":[{"comment":"The paper states that 'Any inconsistent annotations were identified and excluded during the preprocessing stage to ensure high-quality labeling,' but it reports no inter-annotator agreement metric, no exclusion count, and no per-structure disagreement statistic. Since the 45,820 boxes are the sole ground truth for every mAP value in Tables 3–6, this unquantified claim is load-bearing. Please report agreement statistics between the two sonographers (e.g., box-level IoU or Cohen's kappa for structure presence), the number of images and boxes excluded for inconsistency, and per-structure agreement. If the exclusion rate was substantial, the published cohort is a filtered subset and the multi-center difficulty estimates are not representative; if disagreements concentrated on small structures such as NB, NTAPS, MDS, or G, then the low APs in Tables 3–6 conflate label noise with detection difficulty.","section":"§3.2"},{"comment":"The annotation format is described as boxes 'with four vertices per structure,' while all baselines are implemented in MMDetection, which consumes axis-aligned boxes. The paper never states whether mAP-50 is computed on the raw quadrilaterals, on their axis-aligned bounding rectangles, or on a rotated-box representation, nor which conversion is applied before evaluation. This ambiguity affects the definition of every metric in Tables 3–6. Please specify the exact evaluation convention, release the conversion/evaluation code, and, if rotated boxes are used, explain how they are reconciled with the axis-aligned detectors that produce the reported numbers.","section":"§3.2 and §4.2"},{"comment":"The claim that FUSEP is 'the first publicly available dataset and benchmark for fetal early pregnancy ultrasound screening' is a central novelty claim, but the comparison in Table 1 covers only three adult-echocardiography datasets and does not systematically consider public fetal ultrasound datasets from the same or adjacent settings, such as SonoNet standard planes [2], FPUS23 [65], PSFHS [9], and the fetal head biometry dataset [1]. Although those may not be exact early-pregnancy multi-structure detection benchmarks, the 'first' claim should be supported by a broader, dated comparison table or by a clear scope definition that states why these works fall outside the claimed novelty. As written, the claim is stronger than the evidence provided.","section":"Abstract and §1"}],"minor_comments":[{"comment":"The sentence describing mAP-50 as 'the detection results with different thresholds of Intersection over Union (IoU) in the Non-Maximum Suppression (NMS)' is inaccurate; mAP-50 is mean average precision at a single IoU threshold of 0.5. Please correct the description.","section":"§4.2"},{"comment":"The training-set description is ambiguous: 'we use all training sets (612 images of CRL and NT view in Hospital-1, 590 images of CRL view and 607 images of NT view in Hospital-2...)' does not clearly state whether Hospital 1 contributes 612 images per view or 612 total, and no train/validation/test split sizes are given. Please list the exact split sizes for each view and hospital.","section":"§4.2"},{"comment":"The case-to-image counts (766/759/989 cases vs 1,532/1,496/989 images) imply that Hospitals 1 and 2 contribute roughly two images per case while Hospital 3 contributes one; the text should state whether a case can yield multiple retained images and why the ratio differs across hospitals.","section":"§3.1"},{"comment":"Several anatomical structure names are inconsistently capitalized or misspelled, for example 'RhomBencePhalonRBP' and 'Dience Phalon DP'; please standardize these to the medical terms used in the text and the appendix.","section":"Table 2 and Figure A2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about annotation quality and the unspecified quadrilateral-to-axis-aligned conversion is well founded and lands directly on the central benchmark claims. I do not see grounds for rejection if these points are addressed. The paper also relies heavily on citations to the authors' own prior work for several baseline methods; this is not inappropriate given the topic, but the 'first benchmark' claim should be checked against a broader set of fetal ultrasound resources before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new resource — to my knowledge the first public multi-center benchmark for early-pregnancy fetal ultrasound detection — but the annotation reliability is asserted rather than measured, and the box-format/IoU ambiguity leaves the mAP tables provisional.\n\nWhat's actually new: 4,017 CRL and NT images from three hospitals, 45,820 box-level labels across 14 structures, plus standard/non-standard view metadata. That fills a real gap: prior public fetal ultrasound datasets target second/third trimester or cardiac structures, and the closest early-pregnancy work (Lin et al. 2022) stays in-house. The benchmark scaffolding is routine — standard detectors under fully supervised, semi-supervised, UDA, and source-free UDA — which is fine for a resource paper. Headline arithmetic checks out, and the promised GitHub artifacts (code, configs, weights) are the right kind of reproducibility commitment.\n\nSoft spots, in order. First, Section 3.2 says inconsistent annotations were 'identified and excluded' but gives no inter-annotator agreement, no exclusion counts, no per-structure disagreement. Since every mAP in Tables 3–6 depends on those 45,820 boxes, this is load-bearing, not cosmetic. Second, the annotation format: boxes with 'four vertices per structure' while MMDetection baselines output axis-aligned boxes, and the paper never states whether IoU is computed on raw quadrilaterals or after conversion to AABB. That affects the definition of mAP-50 for every row. Third, the appendix calls the dataset 'non-biased' for real-world deployment, but the cohort contains only fetuses without obvious abnormalities; that's an overstatement. Fourth, reproducibility hinges on a GitHub link that cannot be verified from the manuscript; a commit hash and data-access statement would fix that.\n\nThe unmeasured label quality is the weakest link, and it should be quantified before publication, not after. None of this undercuts the core claim: the resource is new, multi-center, and plausibly useful. The problems are fixable in revision. I'd send this to peer review, and I'd cite it if the data actually ships. For a reading group, it's a 'maybe' — the dataset's value outweighs the baseline tables.","headline":"A genuinely new early-pregnancy fetal ultrasound detection benchmark with consistent headline numbers, but the unmeasured annotation reliability and box-format ambiguity make the mAP tables provisional until fixed.","tokens_in":30809,"tokens_out":2709,"would_cite":true,"duration_ms":28719,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FUSEP fills a gap: the first public early-pregnancy fetal ultrasound benchmark, with 4,017 images and 45,820 expert boxes across 14 anatomical structures, plus baselines for semi-supervised and domain-adaptation detection.","keywords":["fetal ultrasound","early pregnancy screening","object detection","benchmark dataset","semi-supervised learning","domain adaptation","crown-rump length","nuchal translucency"],"falsifier":"Re-annotate a random sample of, say, 200 images from both views with two independent experienced sonographers, compute box-level agreement (for example, an IoU-based matching rate or Cohen's kappa on presence or absence of each structure), and compare the result with the dataset's annotations; if agreement is low, or if the exclusion rate reported in preprocessing was high, the benchmark's ground truth is not reliable.","tokens_in":29575,"feed_emoji":"🩺","tokens_out":4693,"duration_ms":50894,"temperature":0.7,"pith_summary":"This paper claims that early-pregnancy fetal ultrasound currently lacks any publicly available dataset, and that the authors close that gap with FUSEP: 4,017 ultrasound images of the crown-rump length (CRL) and nuchal translucency (NT) views collected from three hospitals, carrying 45,820 expert-drawn bounding boxes around 14 anatomical structures. If the claim is correct, researchers gain a reproducible multi-center testbed on which to train and compare detection models, and clinical applications such as standard-plane recognition, image quality control, and cross-device domain adaptation become addressable with shared data rather than private in-house collections. The paper also contributes standardized evaluation protocols, reporting baseline performance under fully supervised, semi-supervised (5% and 10% labels), unsupervised domain adaptation, and source-free domain adaptation settings. The strongest claim is novelty: to the authors' knowledge, this is the first publicly available dataset and benchmark for early-pregnancy fetal ultrasound screening.","feed_headline":"First public benchmark for early-pregnancy fetal ultrasound debuts","feed_subtitle":"45,820 expert boxes over 14 fetal structures, with baselines for scarce labels and cross-device domain shifts.","key_machinery":"The central object is the FUSEP dataset itself: 4,017 images with 45,820 box-level annotations of 14 anatomical structures in the CRL and NT views, plus standardized experimental protocols. The defining design is a multi-center, multi-device collection with detailed annotations of structures including the nasal bone, nuchal translucency, maxilla, mandible, diencephalon, rhombencephalon, head, chest, abdomen, genitalia, and others. What carries the argument is the pairing of these diverse annotations with four training regimes (fully supervised, semi-supervised at two label budgets, unsupervised domain adaptation, and source-free unsupervised domain adaptation), which lets the same images be used to measure model behavior under label scarcity and domain shift.","core_discovery":"The central claim is that FUSEP is the first publicly available dataset and benchmark for early-pregnancy fetal ultrasound screening. The dataset consists of two views recommended by the ISUOG guideline (CRL and NT), spans three hospitals with different devices (SAMSUNG, Sonoscape, GE), and contains 45,820 box-level annotations of 14 anatomical structures on 4,017 images. The paper shows that state-of-the-art object detectors (nine fully supervised, six semi-supervised, six unsupervised domain adaptation, four source-free unsupervised domain adaptation) behave measurably differently across hospitals, and that tiny structures such as the nasal bone (occupying only 0.2% to 0.5% of the detection area) are consistently the hardest to detect, which the authors interpret as a signature of the unique difficulty of early-pregnancy ultrasound. They also identify that cross-center performance drops correlate with changes in imaging devices, making the dataset a natural stress test for domain adaptation methods.","pith_inferences":["A natural next step the authors list is adding segmentation masks; if labels are extended to pixel level, automated measurement of crown-rump length and nuchal translucency thickness becomes feasible.","The label-quality premise is untested: the paper asserts but does not measure inter-annotator agreement, so the stability of the 45,820 boxes should be verified before treating the benchmark's numbers as gold.","Because per-class average precision is reported, practitioners can weight classes by clinical importance (for example, nasal bone absence as a Down-syndrome marker) rather than relying on aggregate mean average precision."],"forward_implications":["Researchers gain a shared benchmark: any detection model can be trained and evaluated on FUSEP and compared against the reported baselines.","Semi-supervised methods can be tested under realistic scarcity: with only 5% or 10% of labels, the dataset measures whether unlabeled data helps early-pregnancy detection.","Cross-device domain shifts become quantifiable: because images come from three hospitals using different ultrasound devices, unsupervised and source-free domain adaptation methods can be compared on a controlled, realistic gap.","Downstream clinical tasks such as standard view recognition, image quality control, and missing-structure diagnosis become researchable on top of the detection baselines.","The documented challenges of extreme scale variation, class imbalance, and feature sparsity set explicit difficulty axes for future methods to report against."],"supporting_citations":[{"why":"ISUOG Practice Guidelines for the 11–14-week scan, which define the CRL and NT views and the scanning standards the dataset follows.","marker":"[3]"},{"why":"Global report on birth defects, quantifying the burden that motivates early screening and automated assistance.","marker":"[12]"},{"why":"Routine first-trimester screening study reporting a 43.1% structural malformation detection rate, which motivates the need for better tools.","marker":"[31]"},{"why":"Prior multi-center study detecting 9 anatomical structures in early pregnancy; the closest prior work, highlighting the gap FUSEP fills.","marker":"[32]"},{"why":"Adult cardiac ultrasound dataset (CardiacUDA) used in the comparison table to show that existing public ultrasound datasets are not early-pregnancy fetal data.","marker":"[73]"},{"why":"Adult echocardiography dataset (CAMUS) used in the comparison table to show the same gap.","marker":"[25]"},{"why":"Large adult echocardiography dataset (EchoNet) used in the comparison table to show that no early-pregnancy fetal dataset publicly exists.","marker":"[46]"}],"fun_headline_variants":["First public fetal ultrasound benchmark debuts with 45k boxes and 14 structures","Early-pregnancy ultrasound benchmark debuts with 45,820 annotations across 3 hospitals","Multi-center fetal ultrasound benchmark targets scarce labels and domain shifts","First open benchmark for early fetal anatomy screening launches with 4,017 images","Fetal ultrasound dataset spans 3 hospitals, 14 structures, and 45k boxes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The expert annotations are treated as correct, but the paper never measures how often two sonographers agree on a box or how many images were discarded as inconsistent, so the ground truth, and every number built from it, could be unstable.","fun_headline_variants_meta":{"raw":{"variants":["First public fetal ultrasound benchmark debuts with 45k boxes and 14 structures","Early-pregnancy ultrasound benchmark debuts with 45,820 annotations across 3 hospitals","Multi-center fetal ultrasound benchmark targets scarce labels and domain shifts","First open benchmark for early fetal anatomy screening launches with 4,017 images","Fetal ultrasound dataset spans 3 hospitals, 14 structures, and 45k boxes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000984,"raw_usage":{"total_tokens":4226,"prompt_tokens":1050,"completion_tokens":3176,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":3073}},"tokens_in":666,"tokens_out":3176,"duration_ms":25322,"temperature":1.0,"reasoning_tokens":3073,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:09:24.399500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of, say, 200 images from both views with two independent experienced sonographers, compute box-level agreement (for example, an IoU-based matching rate or Cohen's kappa on presence or absence of each structure), and compare the result with the dataset's annotations; if agreement is low, or if the exclusion rate reported in preprocessing was high, the benchmark's ground truth is not reliable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Routine first-trimester screening study reporting a 43.1% structural malformation detection rate, which motivates the need for better tools."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior multi-center study detecting 9 anatomical structures in early pregnancy; the closest prior work, highlighting the gap FUSEP fills."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adult cardiac ultrasound dataset (CardiacUDA) used in the comparison table to show that existing public ultrasound datasets are not early-pregnancy fetal data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adult echocardiography dataset (CAMUS) used in the comparison table to show the same gap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large adult echocardiography dataset (EchoNet) used in the comparison table to show that no early-pregnancy fetal dataset publicly exists."}],"review_version":1}