{"id":"bd97f885-03a8-4ae2-a55d-25f904ad991c","arxiv_id":"1908.11624","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Semi-supervised consistency training improves distinct fetal ultrasound anatomy classes but is hurt by a background class and can increase confusion among similar cardiac views.","lead":"This paper tests semi-supervised learning on fetal ultrasound images and finds that adding a hard background class can make semi-supervised training worse than simply using the labeled data. It also finds that semi-supervised learning helps easy-to-distinguish classes like brain and femur but can increase confusion among very similar heart views.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Background-detriment claim is confounded: SSL hyperparameters (cardiac mask threshold 0.25, log TSA) are tuned on the no-background 20-label case and then transferred to the with-background case, so Figure 3's blue bars may fail simply from mistuned hyperparameters.","rationale":"The reader's weakest assumption correctly flags test-set hyperparameter selection and missing variance estimates. My stress-test sharpens this into a specific confound: the paper's headline qualitative result about the background class is based on SSL hyperparameters tuned in the no-background condition. Since the confidence mask threshold for cardiac classes is explicitly set to a value that is optimal for a 13-class softmax without background, applying it to a 14-class problem with background is unjustified. This could reverse the background-detriment conclusion, making the concern more than a quantitative margin issue. However, the paper's other claim—that SSL helps distinct classes at the expense of similar cardiac classes—is partially supported by confusion matrices and is a plausible empirical pattern; the role of the cardiac threshold in producing that pattern is acknowledged. The appropriate remedy is conditional acceptance with a required re-evaluation, exactly matching the reader's verdict, so I leave the verdict unchanged. I credit the paper for a clear experimental setup and for openly reporting the tuning searches, but the missing validation split and code/data availability limit independent verification.","tokens_in":5213,"tokens_out":4627,"duration_ms":44033,"concrete_test":"Rerun the with-background SSL condition at each label count (5, 20, and 50 per class) with the cardiac confidence threshold re-tuned on a held-out validation split from the training set, testing at least thresholds {0.75, 0.5, 0.25, 0.125} while keeping other UDA components fixed; also run each setting with 3 seeds. If every retuned threshold still leaves SSL below the supervised baseline on the test set, the background-detriment claim survives. If any threshold restores parity or improvement, the Figure 3 blue-bar comparison is an artifact of transferred hyperparameters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim \"inclusion of a challenging background class can be detrimental [to SSL]\" rests on Figure 3's blue bars. But the SSL configuration used there is selected in Tables 1 and 2 exclusively from experiments at 20 labelled images per class without background. In particular, Table 1 tunes the cardiac confidence-mask threshold to 1/3 of ηcbm = 0.25 in a 13-class no-background task; Table 2 selects log TSA and Adam in the same setting. The paper then states: \"The best found configuration is then used with 5, 20, and 50 labelled images per class, with and without the background class.\" Adding the background class changes the output distribution from 13 to 14 classes and introduces a highly populated, diverse category, so the optimal masking threshold and TSA schedule need not transfer. The comparison therefore conflates \"background hurts SSL\" with \"background-incorrect hyperparameters hurt SSL.\" The other qualitative result—SSL helps distinct classes while increasing confusion among similar cardiac views—is also read from a single test-tuned threshold, and no variance estimates are reported. These issues are fixable, but they mean the strongest claim is not yet established by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies semi-supervised learning (consistency training / UDA) for standard-plane classification in fetal 2D ultrasound, using the SonoNet architecture and a dataset of 13 anatomical classes plus a background class. With 5, 20, and 50 labelled images per class, the authors report that supervised baselines are surprisingly strong, that SSL improves accuracy on distinct classes but can increase confusion among similar cardiac classes, and that including a challenging background class can make SSL detrimental. The SSL configuration is selected on the 20-labelled-per-class, no-background setting and then applied to all other conditions. The paper is positioned as a response to calls for realistic evaluation of SSL.","tokens_in":5491,"tokens_out":4699,"duration_ms":41020,"significance":"If the findings hold, the paper provides a valuable, realistic counterpoint to benchmark-driven SSL results, showing that class imbalance, a diverse background class, and high inter-class similarity can change the direction of SSL benefits. The use of a real clinical dataset, comparison with a strong supervised baseline, and confusion-matrix analysis are genuine strengths. The central claims, however, rest on test-set hyperparameter selection and single-run results, so the quantitative and qualitative conclusions are not yet established. With additional experiments and proper validation, the paper could make a useful contribution to the medical-imaging SSL literature.","major_comments":[{"comment":"The central claim that including a background class makes SSL detrimental is not established because the SSL hyperparameters are selected on test accuracy from the 20-labelled-per-class, no-background setting and then applied to all other conditions. Table 1 selects the cardiac confidence threshold 1/3*eta_cbm = 0.25 from that setting, and Table 2 selects the log TSA schedule with Adam from the same setting; the text then states that the best found configuration is used for 5, 20, and 50 labelled images per class, with and without the background class. Adding the background class changes the output distribution from 13 to 14 classes and introduces a large, diverse, class-imbalanced category, so the optimal masking threshold and training schedule may not transfer. The comparison in Figure 3 therefore conflates 'background hurts SSL' with 'background-incorrect hyperparameters hurt SSL'. A proper validation split or per-condition hyperparameter selection is needed.","section":"Section 4, Tables 1 and 2, Figure 3"},{"comment":"No variance estimates, multiple seeds, or significance tests are reported anywhere. Every accuracy number is the result of a single training run, and several differences used to support the conclusions are small (for example, overall 0.754 vs 0.720 in Table 2, and the blue-versus-red gaps in Figure 3). These differences could be within run-to-run variation, so the direction and magnitude of the claimed SSL benefit is not established. The authors should rerun experiments with several seeds and report mean and standard deviation.","section":"Section 4, Tables 1 and 2, Figure 3"},{"comment":"The qualitative claim that SSL helps distinct classes while increasing confusion among similar cardiac views is read from a single pair of confusion matrices and one threshold choice. The cardiac threshold of 0.25 was itself chosen to avoid harming cardiac classes in Table 1, so the confusion-matrix pattern is conditional on that test-selected hyperparameter. The authors should show how the confusion pattern varies across thresholds and seeds before drawing this conclusion.","section":"Section 4 and Figure 4"}],"minor_comments":[{"comment":"The class label 'SAGITAL' should be 'SAGITTAL', and the caption contains the same misspelling in 'SPINE-SAGITAL'.","section":"Figure 1"},{"comment":"There is a typo: 'baslines' should be 'baselines'.","section":"Figure 3 caption"},{"comment":"The threshold notation is inconsistent between the table and the text: the table lists '1/3*eta_cbm', while the text says 'A cardiac threshold of 0.25' and '1/3*eta_cbm (0.25)'. State explicitly that eta_cbm = 0.75 in all experiments to make the arithmetic clear.","section":"Table 1 and Section 4"},{"comment":"Clarify how the reported 'overall' accuracy is computed when the background class is excluded: is it mean per-class accuracy or overall accuracy over the non-background test subset? This affects interpretation of the numbers.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important practical question and the negative result would be useful if confirmed. The main risk is that test-based hyperparameter selection and lack of variance estimates overstate the findings; this is fixable with additional experiments. If the authors cannot provide a validation-based selection procedure and repeated-seed results, I would not support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth knowing about: it tests consistency-based SSL (Pi model and UDA) on a realistic fetal ultrasound classification task and reports two things. First, SSL mostly helps classes that are already distinct and can increase confusion among very similar cardiac views. Second, adding a challenging background class to the unlabeled pool can make SSL worse than a supervised baseline. The first result is credible and nicely visualized in the confusion matrices. The second is the paper's headline, and it is not actually established by the reported experiments.\n\nThe problem is that the SSL configuration is tuned on the 20-labels-per-class no-background case. Table 1 picks a cardiac confidence threshold of 0.25; Table 2 picks log TSA with Adam. Then the paper says it uses that configuration for all label counts, with and without background. Adding the background class changes the output distribution from 13 to 14 classes and introduces a large, diverse category. There is no reason to expect the tuned threshold or TSA schedule to transfer well. The comparison therefore conflates \"background hurts SSL\" with \"background-inappropriate hyperparameters hurt SSL.\" The authors' own justification that 0.25 is reasonable for four similar cardiac classes makes sense without background, but it does not address how background changes the confidence distribution. This is a fixable flaw: rerun the tuning on the with-background case, or at least do a small sensitivity sweep, and report whether the conclusion holds.\n\nThe paper also gives no standard deviations or multiple seeds. All results appear to be single runs. For a paper whose main quantitative claim is about a 5–10 point accuracy difference, that is a real gap. The qualitative pattern in the confusion matrices is less affected by this, but the accuracy bars in Figure 3 should be treated as indicative, not definitive.\n\nWhat is genuinely new is the application domain and the observation that SSL's benefit is asymmetric across class similarity. That is a useful contribution to the medical-imaging SSL literature, and it aligns with Oliver et al.'s call for realistic evaluation. The paper does not ship code or data, which limits independent verification, and the citation pattern is fine—no obvious self-citation inflation.\n\nI would accept this for peer review because the question is important and the empirical setup is relevant, even though the main claim needs a cleaner evaluation protocol before publication. If the authors fix the tuning issue and add variance estimates, this becomes a solid subfield-level measurement. For now, it is a cautionary result with a confound at its core.","headline":"Useful cautionary study on SSL in fetal ultrasound, but the headline claim about background classes is confounded by test-set hyperparameter tuning and missing variance estimates.","tokens_in":5990,"tokens_out":1374,"would_cite":false,"duration_ms":14405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semi-supervised learning mostly benefits distinct classes in fetal ultrasound and can be detrimental when a background class is included, especially for similar cardiac views.","keywords":["semi-supervised learning","fetal ultrasound","standard plane classification","consistency training","UDA","cardiac view confusion","background class","SonoNet"],"falsifier":"Run the 5/20/50-label protocol again but choose the cardiac threshold and TSA schedule on a held-out validation split for each label count, or fix all hyperparameters before seeing test labels; if SSL then fails to beat the supervised baseline on most label counts, or if the background class no longer causes a drop, the central claim would be weakened.","tokens_in":5030,"feed_emoji":"🩺","tokens_out":6622,"duration_ms":54555,"temperature":0.7,"pith_summary":"This paper tries to establish whether semi-supervised learning (SSL), which has worked on clean benchmark image sets, still helps when the data are messy fetal ultrasound frames. The authors find that a supervised Sononet baseline already classifies standard planes surprisingly well from as few as 20 labelled images per class. Adding a consistency-based SSL loss helps classes that are already visually distinct, such as brain, femur, kidney, and lips, but can increase confusion among the four very similar cardiac views. A large, heterogeneous background class in the unlabeled pool tends to make SSL worse than the supervised baseline. If these results are right, SSL cannot be applied uniformly in fetal ultrasound; it needs per-class treatment and careful handling of out-of-distribution frames.","feed_headline":"Semi-supervised learning can hurt similar classes in fetal ultrasound","feed_subtitle":"Consistency training helps distinct classes but increases cardiac confusion; background frames make it worse.","key_machinery":"The central mechanism is a consistency loss $L_{KL}(x_u,w)=KL(f(x_u;w)\\|f(x_u';w))$ that uses the softmax prediction of an unlabelled image as the target for the same image under random augmentation, weighted at $\\lambda=0.5$ and combined with training signal annealing, confidence-based masking (default $\\eta_{cbm}=0.75$), entropy minimization, and softmax temperature 0.8, a combination the paper calls UDA. The supervised backbone is Sononet, a 15-layer CNN with global average pooling. The decisive control in the argument is the lowered cardiac confidence-mask threshold of $\\eta_{cbm}/3=0.25$, since this single change decides whether SSL helps or hurts the cardiac classes.","core_discovery":"The central claim is that SSL's benefit in fetal 2D ultrasound standard-plane classification is class-dependent and can be negative. Without the background class and with a cardiac-specific confidence-mask threshold of 0.25, the best UDA configuration improves both overall accuracy (0.720 to 0.754) and grouped cardiac accuracy (0.868 to 0.936) at 20 labels per class; inclusion of the background class makes SSL accuracy fall below the supervised baseline for most label counts. Confusion matrices show that SSL drives distinct classes toward fully-supervised performance while increasing mislabelling among the four cardiac views, implying that unlabeled data helps only when classes are already separable.","pith_inferences":["Not shown in the paper: the test-set-selected hyperparameters, especially the cardiac threshold of 0.25 and the log-TSA-with-Adam configuration, likely inflate the reported SSL margin; a validation-split protocol would probably show a smaller benefit.","Not shown in the paper: a natural testable extension is to make the confidence threshold class-dependent, set near one over the number of classes in each group, and check whether SSL then helps similar classes.","Not shown in the paper: the background-class result suggests that explicit out-of-distribution detection or filtering of confusable background frames could turn detrimental unlabeled data into useful signal.","Not shown in the paper: the per-class pattern predicts that SSL will also hurt other high-similarity medical-image groupings, such as different views of the same organ, when unlabeled data come from the same heterogeneous distribution."],"forward_implications":["With 20 labelled images per class and no background class, the best UDA configuration raises overall accuracy from 0.720 to 0.754 and grouped cardiac accuracy from 0.868 to 0.936, but only when the cardiac confidence mask threshold is lowered to 0.25.","Including the background class in the unlabeled set makes SSL fall below the supervised baseline in most of the 5/20/50-label settings, so unlabeled frames that resemble standard planes but are not standard planes are not harmless.","In the 20-label case, SSL lifts distinct classes such as brain, femur, kidney, and lips from mid-0.80 to mid-0.90 accuracy while increasing confusion among the four cardiac views.","A supervised baseline already reaches roughly 70% overall and 80% grouped cardiac accuracy from 20 labels per class, and accuracy plateaus quickly, so the realistic margin SSL can add is small and concentrated in specific classes."],"supporting_citations":[{"why":"Supplies the Sononet architecture and the fetal standard-plane classification task used as the supervised baseline and backbone.","marker":"[2]"},{"why":"Defines the consistency-loss family (Pi model) that the paper adapts as its SSL objective.","marker":"[9]"},{"why":"Provides the UDA combination of TSA, confidence masking, entropy minimization, and temperature whose behavior on fetal data is the subject of the study.","marker":"[14]"},{"why":"Is the call for realistic SSL evaluation that the paper directly tests against fetal ultrasound.","marker":"[10]"},{"why":"Contributes the entropy-minimization term included in the SSL objective.","marker":"[7]"}],"fun_headline_variants":["Semi-supervised learning: distinct fetal classes win, similar lose","Fetal ultrasound: SSL helps separable classes, hurts similar ones","Background class derails semi-supervised fetal anatomy learning","Semi-supervised learning backfires on similar fetal ultrasound classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hyperparameters chosen on the test set for the 20-labelled-per-class case, especially the cardiac confidence threshold of 0.25 and the log-TSA-with-Adam configuration, transfer to the 5- and 50-label runs; if they do not, the size or direction of the claimed SSL benefit is not established.","fun_headline_variants_meta":{"raw":{"variants":["Semi-supervised learning: distinct fetal classes win, similar lose","Fetal ultrasound: SSL helps separable classes, hurts similar ones","Background class derails semi-supervised fetal anatomy learning","Semi-supervised learning backfires on similar fetal ultrasound classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2722,"prompt_tokens":725,"completion_tokens":1997,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":341,"completion_tokens_details":{"reasoning_tokens":1927}},"tokens_in":341,"tokens_out":1997,"duration_ms":14432,"temperature":1.0,"reasoning_tokens":1927,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:09:05.818713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 5/20/50-label protocol again but choose the cardiac threshold and TSA schedule on a held-out validation split for each label count, or fix all hyperparameters before seeing test labels; if SSL then fails to beat the supervised baseline on most label counts, or if the background class no longer causes a drop, the central claim would be weakened.","supporting_citations":[{"cited_title":"IEEE Transactions on Medical Imaging 36(11), 2204–2215 (2017)","cited_arxiv_id":null,"evidence_quote":"Supplies the Sononet architecture and the fetal standard-plane classification task used as the supervised baseline and backbone."},{"cited_title":"ICLR (2017)","cited_arxiv_id":null,"evidence_quote":"Defines the consistency-loss family (Pi model) that the paper adapts as its SSL objective."},{"cited_title":"NeurIPS (2018)","cited_arxiv_id":null,"evidence_quote":"Is the call for realistic SSL evaluation that the paper directly tests against fetal ultrasound."},{"cited_title":"NeurIPS (2005)","cited_arxiv_id":null,"evidence_quote":"Contributes the entropy-minimization term included in the SSL objective."}],"review_version":1}