{"id":"7fc617e5-8747-4d92-b333-79056ff33fb5","arxiv_id":"2412.15967","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Self-supervised contrastive learning on 48,434 skeletal radiographs achieves 96.6% accuracy in anatomical region classification and can detect incorrect PACS labels.","lead":"Self-supervised contrastive learning classifies the anatomical region of skeletal X-ray images with high accuracy, even when hospital labels are noisy. The method also flags data entry errors and could help hospitals clean their imaging metadata and use external data in low-resource settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set leakage in self-supervised pretraining likely inflates reported accuracies; Figure 1 shows 'all data' used for pretraining.","rationale":"The reader's verdict of CONDITIONAL is appropriate, but the most load-bearing concern is not the one highlighted in the reader's weakest_assumption. The selection-bias issue with the corrected accuracy (auditing only the 328 model errors and ignoring the ~9,400 agreements) is a real and valid statistical flaw, and it directly undermines the 'theoretical accuracy' of 98.0%/98.8%. However, that critique applies to a secondary, post-hoc correction of the test labels. The more fundamental problem is that the main evaluation numbers—96.6% and 97.7%—are themselves suspect because the self-supervised backbone was pretrained on all data, including the test set, as indicated by Figure 1 and Section 3.3. If the backbone has seen the test images, the linear evaluation is no longer a measure of generalization, and all downstream analyses (including the error audit) are biased. This is a standard methodological requirement in self-supervised learning: the unlabeled pretraining corpus must not include the test set. The reader's rationale does mention 'test-set pretraining appears to leak information' but does not elevate it to the primary weakest assumption. The concrete test I propose would settle the matter definitively: rerun pretraining on the training split only and observe whether the reported accuracies hold. Given the authors did not release code or data, this retraining is necessary before the quantitative claims can be trusted. The conditional verdict remains correct because the paper presents a plausible method and the qualitative findings (e.g., that contrastive pretraining helps in low-label regimes, that spine/shoulder labels are noisy) could survive a clean rerun, but the specific numbers should be re-estimated.","tokens_in":8512,"tokens_out":3452,"duration_ms":31630,"concrete_test":"Retrain SimCLR and BYOL from scratch on only the 31,011 training images (excluding the 7,677 validation and 9,746 test images), keeping all hyperparameters identical, then train the linear probe on the same labeled training subsets and evaluate on the original test split. If the single-model accuracy drops by more than 1 percentage point from 96.6%, or if the low-label 1% result falls below 90%, the reported results are inflated by test-set leakage. Additionally, count the number of test errors; if the error set shrinks substantially when the test set is excluded from pretraining, the label-noise analysis should be redone on a clean held-out split.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claims—96.6% single-model and 97.7% ensemble accuracy, plus the 92.2% low-label result—rest on a clean evaluation protocol. That protocol is violated: Figure 1 explicitly states 'Self-Supervised Pre-training (all data)', and Section 3.3 describes pretraining on the 48,434 radiographs without ever restricting it to the 31,011-image training split introduced in Section 3.1. Unsupervised pretraining on the test set is transductive learning: the backbone sees each test image before the linear probe is trained. This can inflate linear-evaluation accuracy because the feature extractor has already adapted to the test distribution, and it makes the comparison against the fully supervised baseline (trained only on labeled subsets) unfair. It also contaminates the label-noise audit: the 328 'test errors' are generated by a model that has seen the test images, so the error set is not a valid sample of held-out behavior. The paper's recommendation not to unfreeze the backbone (Section 3.3) suggests the authors are aware that the linear head alone is a fair evaluator, but that does not fix the backbone leakage. If the backbone is pretrained on all data, the reported numbers are not estimates of generalization to new hospital data, undermining the 'ready for real-world application' conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses automatic classification of skeletal radiographs into 14 anatomical regions from a real-world PACS dataset of 48,434 images. The authors pretrain ResNet18 backbones with SimCLR, BYOL, and supervised contrastive learning, then evaluate frozen-backbone linear probes. They report a single-model test accuracy of 96.6%, an ensemble accuracy of 97.7%, and 92.2% accuracy using only 1% of the training labels. They also report an expert-radiologist audit of the single model's 328 test errors, finding that 35% of these were incorrect PACS labels and 11% were out-of-domain images, which leads to a claimed 'theoretical' accuracy of 98.0% (single) and 98.8% (ensemble) after relabeling. The paper further proposes using the model to detect and correct noisy PACS metadata.","tokens_in":8800,"tokens_out":2897,"duration_ms":27305,"significance":"If the reported numbers hold under a clean evaluation protocol, the work would be practically valuable: it demonstrates that self-supervised representation learning can enable anatomical-region classification with very few labels and that learned models can flag data-entry errors in hospital PACS systems. The authors deserve credit for using a large in-house clinical dataset, for explicitly targeting label noise as a research question, and for obtaining independent expert-radiologist labels to ground the noise analysis. However, the central quantitative claims are weakened by a potentially serious evaluation-protocol issue: the self-supervised pretraining appears to include the test set as unlabeled data, and the supervised contrastive model appears to use test labels during pretraining. This affects the headline accuracies, the low-label results, and the error audit, and it must be resolved before the results can be accepted as estimates of held-out generalization.","major_comments":[{"comment":"The pretraining protocol appears to leak test-set information. Section 3.1 defines a 64/16/20 train/validation/test split of the 48,434 radiographs, but Section 3.3 states that pretraining is performed on the full 48,434 radiographs without excluding the test split, and Figure 1 explicitly labels 'Self-Supervised Pre-training (all data)'. Moreover, the supervised contrastive method is described as using label information 'for the whole dataset', meaning test labels are used during pretraining for the SupCon model that contributes to the ensemble. Under this protocol, the reported 96.6% single-model and 97.7% ensemble accuracies are transductive estimates, not held-out generalization, and the comparison against the fully supervised baseline trained only on labeled training subsets is unfair. The authors must rerun the pretraining using only the 31,011-image training split (and the validation split if needed, with the test split strictly held out) and report whether the main conclusions change.","section":"Section 3.3 and Figure 1"},{"comment":"The corrected 'theoretical' accuracy is not established because the expert audit is conditioned on the model's disagreements. The radiologist audited only the 328 test images where the SimCLR prediction differed from the PACS label, leaving the roughly 9,400 test images where model and PACS label agreed unaudited. Errors in which both the model and the PACS label are wrong would not appear in the 328 cases, so the claim that correcting labels raises accuracy to 98.0% (and 98.8% for the ensemble) is an upper bound, not a measured value. To support the data-cleaning claim, the authors should audit a random sample of the agreement cases (or an independent random sample of the full test set) and report the actual error rate after relabeling. This is a load-bearing issue for the paper's stated application of correcting PACS metadata.","section":"Section 4.4 and Table 2"},{"comment":"The low-label experiment inherits the same test-set leakage: the SimCLR and BYOL backbones were pretrained on the test images, so the 92.2% accuracy at 1% of training labels reflects features that have already seen the test distribution. Additionally, the supervised baseline is described only as a 'randomly initialized ResNet18 trained fully supervised'; the paper does not specify whether this baseline uses the same image cleaning, gauge augmentation, and SimCLR-style augmentations, or the same optimization schedule. Without these details and without a leakage-free pretraining, the large gap between the self-supervised and supervised curves cannot be attributed to self-supervision. Please repeat the low-label sweep with a backbone pretrained only on the training split and report the baseline setup precisely.","section":"Section 4.1 and Figure 2"}],"minor_comments":[{"comment":"The sentence 'accordingly, we set τ_base = 0.9995 as recommended in [8]' refers to BYOL's exponential moving average decay, not to the temperature τ used in SimCLR; using the same symbol for both quantities is confusing. Please rename the BYOL EMA parameter (e.g., m or target_ema) and clarify the values.","section":"Section 3.3 (BYOL)"},{"comment":"The confusion matrices in Figure 4 are difficult to read: diagonal entries are omitted in panel (a), and the delta matrix in panel (b) uses colors without a numeric legend for the magnitude of changes. Please add clear numeric annotations and a caption explaining how to read the 'delta' matrix.","section":"Section 4.4 and Figure 4"},{"comment":"The row 'Gauge Occurrences 1/3 probability for 0, 1, 2' is ambiguous: it should state explicitly that with probability 1/3 each, zero, one, or two gauges are inserted, and whether the same distribution is used in pretraining and training.","section":"Supplementary Table 1"},{"comment":"The sentence 'we add a novel augmentation where we use 6 example gauges extracted from the dataset' should clarify whether those 6 example gauges are extracted from the training split only or from the entire dataset, since using test images to build augmentations would be another form of leakage.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The test-set leakage is the decisive issue: if the authors cannot show that the central results survive retraining on the training split only, the paper's main claims do not hold as stated. The paper's framing as a 'real-world ready' system also makes the leakage more serious, because the claimed deployment scenario is exactly the held-out generalization setting. I would ask the editor to require the rerun as a condition of any further consideration. The audit-selection issue in Section 4.4 is also important, but it is a fixable experimental-design gap rather than an unfixable flaw. The paper should also cite and compare with prior work on label-noise auditing in medical imaging, as the related work section is thin on this specific point."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look for one reason: the expert radiologist audit of model failures is a genuinely useful way to expose noisy PACS metadata, and the gauge augmentation is a thoughtful, targeted fix for a real shortcut. Those contributions stand on their own. The problem is that the headline numbers—96.6% single-model, 97.7% ensemble, and the 92.2% 1% label result—are not trustworthy because the self-supervised backbone appears to be pretrained on the full 48,434-image dataset, including the test split. Figure 1 makes this explicit: “Self-Supervised Pre-training (all data).” Section 3.1 defines the train/validation/test split, but Section 3.3 describes pretraining on all radiographs without restricting to the training split. That is transductive learning: the feature extractor has already seen the test images before the linear probe is trained. It inflates linear evaluation accuracy and makes the comparison against the fully supervised baseline, which only trains on labeled subsets, unfair. The low-label comparison inherits the same problem. The corrected accuracy of 98.0% is also selection-biased: the expert only audited the 328 images where the model disagreed with the PACS label, not the thousands where both agreed. Both-wrong agreements would lower the corrected number. I don't think these are deliberate deceptions—the paper is transparent enough about the all-data pretraining—but they are load-bearing flaws. The method probably works, and the label-noise detection idea is solid, but the empirical claims as reported are not yet supported. There are also minor issues: no error bars, single-center data, no code or data release, and the low-label experiment lacks repeated-seed variance. The paper deserves a serious referee because the question—how clean is real-world PACS data, and can self-supervised models catch the dirt—is important for operational medical AI, and the audit procedure is a legitimate contribution. But the evaluation protocol has to be corrected: retrain the backbone on the training split only, re-report all numbers, and ideally validate on an external dataset. If the numbers hold up after that, this becomes a useful practical paper. As is, I would not cite the headline accuracy; I might cite the audit methodology.","headline":"A practically useful label-noise audit undermined by a test-set leakage that makes the headline accuracies unverifiable.","tokens_in":9293,"tokens_out":2296,"would_cite":false,"duration_ms":20469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that contrastive self-supervised pretraining on 48,434 skeletal radiographs classifies 14 anatomical regions at 96.6% accuracy with one model, 97.7% with an ensemble, and uses the model's disagreements to catch wrong PACS…","keywords":["self-supervised learning","contrastive learning","radiograph classification","anatomical region classification","PACS metadata quality","label noise","SimCLR","medical imaging"],"falsifier":"Randomly sample a few hundred test-set images in which the model and the PACS label agree, have an expert radiologist independently re-label them, and recompute accuracy on the re-labeled set; if a substantial share of these agreements are actually mislabeled by both PACS and the model, the theoretical 98.0% accuracy does not hold, whereas an error rate near zero would confirm it.","tokens_in":8326,"feed_emoji":"🩻","tokens_out":8496,"duration_ms":62996,"temperature":0.7,"pith_summary":"The paper claims that self-supervised contrastive pretraining on 48,434 skeletal radiographs is enough to classify the anatomical region into 14 classes with 96.6% accuracy using a single model and 97.7% using an ensemble. It also claims that only 310 labeled images (1% of the training set) suffice to reach 92.2% accuracy, making the approach usable in low-resource settings. The central application is data cleaning: an expert radiologist's review of the model's 328 test errors found that 35% of them were wrong PACS labels and 11% were out-of-domain images, and 98 of the 116 wrongly labeled images were actually predicted correctly by the model. Accounting for these label errors raises the corrected accuracy to 98.0% for the single model and 98.8% for the ensemble. If true, this means existing hospital metadata can be audited and corrected from the model's disagreements.","feed_headline":"Self-supervised X-ray model hits 96.6% accuracy, catches bad labels","feed_subtitle":"Just 310 labeled images give 92.2%, and an expert audit found a third of test errors were wrong PACS labels.","key_machinery":"The load-bearing mechanism is contrastive self-supervised pretraining: SimCLR trains a ResNet18 backbone to pull together two augmentations of the same radiograph and push apart other images in a batch, while BYOL removes the negative pairs and trains an online network to predict a target network's representation. A supervised contrastive variant extends positive pairs to all images sharing the same PACS label. After pretraining, the frozen backbone is evaluated with a single linear layer, and an ensemble averages the softmax outputs of the three pretrained models. Two custom preprocessing steps support the claim: border removal and rotation normalization reduce reliance on image-frame artifacts, and a novel augmentation that randomly inserts zero to two surgery-planning gauges at random locations and scales deliberately prevents the model from using gauges as a shortcut, which the paper verifies with gradient-based heatmaps.","core_discovery":"On the paper's own terms, the central discovery is that contrastive self-supervised pretraining on unlabeled skeletal radiographs produces anatomical region features so strong that a single linear layer on top reaches 96.6% test accuracy across 14 classes, and averaging the softmax outputs of three such models (SimCLR, BYOL, and supervised contrastive) reaches 97.7%. The same features are label-efficient: training the linear layer on just 1% of the training labels (310 images) yields 92.2%, far above the 57.1% of a fully supervised baseline trained from scratch. The paper further establishes that the remaining errors are largely label noise rather than model failures: an expert radiologist re-labeling the 328 disagreement cases found 116 incorrect PACS labels, 36 out-of-domain images, and 2 unusable images, with 98 of the 116 incorrect labels being images the model had classified correctly. After filtering and correcting these labels, the theoretical accuracy becomes 98.0% for the SimCLR model and 98.8% for the ensemble. The conclusion the authors draw is that a self-supervised model can be used as a practical tool to find and fix data-entry errors in PACS metadata.","pith_inferences":["The reported corrected accuracy of 98.0% is conditional on the audit being restricted to the 328 cases where the model disagreed with the PACS label; the roughly 9,400 test images where they agreed were never re-examined, so the true accuracy could be lower if some of those labels are wrong in the same way as the model.","The same disagreement-audit procedure could be applied to the training and validation splits, effectively cleaning the entire dataset; the paper notes its architecture allows this but does not carry it out.","Because the gauge augmentation was designed to combat a dataset-specific shortcut, the method's transfer to other radiograph collections may depend on whether similar domain artifacts are handled in the same way.","The claim that the model corrects data-entry mistakes is about agreement between model and label, not about the model being a perfect oracle; a future test could compare the model's corrected labels against a second independent re-labeling to measure agreement."],"forward_implications":["A hospital with only a few hundred labeled radiographs can train a usable anatomical region classifier, since 1% of the labels (310 images) reach 92.2% accuracy.","PACS metadata can be audited by comparing model predictions to stored labels; the spine, shoulder, and clavicle classes are the most inconsistent and would benefit first.","External datasets that arrive without anatomical metadata can be automatically labeled, enabling their inclusion in research cohorts.","Downstream models that are only valid for specific body regions can reject or reroute out-of-domain inputs using the predicted region, reducing silent misdiagnosis.","The ensemble's 97.7% accuracy on raw PACS labels implies that most residual 'errors' in real-world radiograph metadata are label corruption, not model failure."],"supporting_citations":[{"why":"Supplies the SimCLR contrastive objective used for the main self-supervised pretraining.","marker":"[4]"},{"why":"Supplies the BYOL objective, the second self-supervised baseline that reaches the same 96.6% accuracy.","marker":"[8]"},{"why":"Supplies the supervised contrastive loss used as the third training approach in the ensemble.","marker":"[15]"},{"why":"Defines the ResNet18 architecture used as the frozen feature backbone.","marker":"[10]"},{"why":"Provides the medical-imaging self-supervised baseline and the reason for omitting Gaussian blur to preserve fine-grained features.","marker":"[1]"},{"why":"Provides the image preprocessing routines for border removal and rotation normalization.","marker":"[2]"},{"why":"Supplies the T-SNE visualization that illustrates feature separation and supports the label-noise analysis.","marker":"[20]"}],"fun_headline_variants":["Self-supervised X-ray model hits 96.6% and flags bad PACS labels","X-ray model: 96.6% accurate, detects PACS label errors","Only 1% labeled data gives 92.2% X-ray anatomy accuracy","X-ray anatomy model: 96.6% accurate, flags mislabeled PACS images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The corrected accuracy of 98.0% assumes that the 328 images where the model and the PACS label disagreed are the only test-set errors, so fixing those labels fully accounts for label noise; the roughly 9,400 test images where the model and label agreed were never audited.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised X-ray model hits 96.6% and flags bad PACS labels","X-ray model: 96.6% accurate, detects PACS label errors","Only 1% labeled data gives 92.2% X-ray anatomy accuracy","X-ray anatomy model: 96.6% accurate, flags mislabeled PACS images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3162,"prompt_tokens":1051,"completion_tokens":2111,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2020}},"tokens_in":667,"tokens_out":2111,"duration_ms":14131,"temperature":1.0,"reasoning_tokens":2020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:55:03.936077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample a few hundred test-set images in which the model and the PACS label agree, have an expert radiologist independently re-label them, and recompute accuracy on the re-labeled set; if a substantial share of these agreements are actually mislabeled by both PACS and the model, the theoretical 98.0% accuracy does not hold, whereas an error rate near zero would confirm it.","supporting_citations":[{"cited_title":"In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H","cited_arxiv_id":null,"evidence_quote":"Supplies the supervised contrastive loss used as the third training approach in the ensemble."},{"cited_title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp","cited_arxiv_id":null,"evidence_quote":"Defines the ResNet18 architecture used as the frozen feature backbone."},{"cited_title":"2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp","cited_arxiv_id":null,"evidence_quote":"Provides the medical-imaging self-supervised baseline and the reason for omitting Gaussian blur to preserve fine-grained features."},{"cited_title":"Journal of Machine Learning Research 9, 2579–2605 (2008)","cited_arxiv_id":null,"evidence_quote":"Supplies the T-SNE visualization that illustrates feature separation and supports the label-noise analysis."}],"review_version":1}