REVIEW 5 major objections 5 minor 35 references
The central claim is that label noise, not model capacity, limits VCE anomaly detection, and a loss-mixture cleaning pipeline lifts F1 from 54% to 72%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 03:41 UTC pith:SEMKJFPX
load-bearing objection Useful artifact, shaky evidence: the cleaned VCE splits may help the community, but the paper's core claim about detecting mislabels is undercut by biased noise injection and high false positives. the 5 major comments →
Reliable Mislabel Detection for Video Capsule Endoscopy Data
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that per-sample training loss separates clean, hard, and mislabeled samples well enough to drive a full dataset-cleaning workflow. On Kvasir-Capsule with controlled label flipping, the pipeline recovers 456/471 injected mislabels at 1% noise and 2262/2360 at 5%; on the Galar dataset it filters 167,709 samples (4.8%) and corrects 31,650 (0.9%). Training an anomaly detector on the cleaned development set yields 93.83% accuracy and 71.58% F1 on the untouched test set, compared with 54.38% for the best earlier baseline and 37.01% for another. The authors further show that cleaning increases the model's maximum confidence on test predictions (0.85 to 0.96), and that three
What carries the argument
The key mechanism is a three-component Gaussian Mixture Model (GMM) fitted to the per-sample average training loss of a compact convolutional classifier. The component with the highest mean is interpreted as the mislabeled population; a noise-reduction score (the change in noise probability after a label correction) guides which samples to flip, and a final step filters the highest-noise samples. This transforms the model's own loss signal into a ranking of suspected label errors.
Load-bearing premise
The pipeline assumes that the training examples a model finds hardest to learn, measured by the largest losses, are the ones whose labels are wrong, rather than simply being difficult or ambiguous images.
What would settle it
An experiment with a dataset whose labels are all correct but whose samples vary widely in difficulty (e.g., by adding synthetic blur or occlusion to a fraction of clean images) would falsify the core assumption if the GMM's highest-mean component flags those clean-but-hard images as mislabels and the pipeline removes them. Concretely, compare the pipeline's false-positive removal rate on such a difficulty-only dataset against its performance on the same dataset with real injected label noise.
If this is right
- Cleaning the training set before anomaly detection improves F1 on the Galar benchmark by 17 percentage points over the best prior baseline, without changing the test set or the model architecture.
- Models trained on cleaned data produce more confident predictions (maximum confidence 0.85 to 0.96), which is relevant for on-device screening decisions.
- The released corrected and filtered dataset splits give the community a ready-to-use cleaned training set for VCE anomaly detection.
- Because the pipeline is unsupervised and requires only a trained classifier and its loss values, it can be applied to other medical datasets with ambiguous class boundaries.
Where Pith is reading between the lines
- The pipeline's own numbers show a high false-positive rate on clean data: at 1% injected noise, 916 of 1,372 removed samples were actually clean. This suggests part of the F1 gain may come from pruning hard, ambiguous examples rather than from correcting label errors; a controlled comparison against random or uncertainty-based pruning would separate those effects.
- The GMM's three-component assumption couples label noise with sample difficulty. In datasets where the hardest clean examples are as rare as mislabels, the highest-mean component may mix both populations, so the method's performance likely degrades gracefully only if true noise is the dominant source of high loss.
- If the cleaned Galar splits become widely used, baseline comparisons in subsequent VCE anomaly-detection papers will need to state explicitly whether they train on the cleaned or original development set, since the 17-point F1 gap makes the choice decisive.
- A testable extension: running the same pipeline with different classifier capacities and loss functions should shift the GMM separation; measuring how the precision of mislabel detection changes across those shifts would map the method's reliability boundary.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a mislabel-detection pipeline for Video Capsule Endoscopy (VCE) datasets. The pipeline trains a MobileNetV3 classifier, fits a three-component Gaussian Mixture Model (GMM) to per-sample training losses, corrects the k_c samples with the largest noise-reduction score, retrains, and then filters the k_f samples with the highest noise probability. It is evaluated in two stages: (1) a controlled noise-injection experiment on Kvasir-Capsule, and (2) application to the Galar dataset, followed by clinical review of 100 flagged samples by three gastroenterologist co-authors and retraining of an anomaly detector. The authors report F1-score improvements on Galar from 53.70 (uncleaned) to 71.58 (filtered dev set) and to 73.67 when the test set is also filtered, and claim that the framework reliably detects mislabeled data.
Significance. If the claims were sound, the work would be valuable for medical dataset curation: it addresses a real bottleneck in VCE and uses two large public datasets. The resulting cleaned splits could be reused by the community. However, the central claim that the framework 'successfully detects incorrectly labeled data' is not established by the current experiments. The controlled noise-injection protocol is biased toward high-uncertainty samples, the reported false-positive counts are very large, the 'Filtered Filtered' result is circular, and the clinical validation is small and not independent. The paper also leaves the key hyperparameters k_c and k_f unspecified. These are load-bearing issues, not presentation issues.
major comments (5)
- [Section III-C and Table I] The controlled experiment does not test general mislabel detection because the injected noise is not random: label noise was 'introduced primarily by randomly selecting samples from the mid- and high-uncertainty groups.' Since the GMM in Section III-E flags samples by high loss, this selection guarantees that injected noisy samples are already concentrated in the high-loss region. The high recall in Table I (456/471 at 1%) is therefore an artifact of the injection protocol. More importantly, Table I shows 916 clean samples filtered at 1% injected noise, i.e., only about 33% of filtered/corrected samples are genuinely noisy. The pipeline is primarily identifying hard or uncertain examples, not label errors.
- [Table II, Section IV] The 'Filtered Filtered' row evaluates the anomaly detector on a test set that was itself filtered by the same pipeline. The text states that the test set was 'additionally filtered to investigate if this leads to an additional classification enhancement.' This is circular: filtering the test set removes hard or uncertain examples and can inflate precision and F1-score. The row is not comparable to the baselines or to the 'Filtered Uncleaned' row, and the 89.88% precision in that row should not be used to support the claim of improved detection after cleaning.
- [Sections III-B and IV] The number of corrected labels k_c and the number of filtered labels k_f are free parameters of the pipeline, but the manuscript never specifies how they are selected. These parameters directly determine the counts in Table I and the composition of the cleaned Galar dev set. Without a stated selection rule (e.g., based on a validation set or a fixed criterion independent of ground truth), the controlled experiment is not reproducible, and the reported cleaning results could be tuned. This is a load-bearing methodological gap.
- [Section III-F and Clinical Validation] The clinical validation is not representative of the pipeline's actual filtering behavior. The 100 reviewed samples are selected from the top 500 by noise-reduction score, whereas the pipeline filters 167,709 samples from Galar. Precision@100=78% on this hand-picked subset does not measure precision on the filtered set, and Table I shows that filtered sets contain many clean samples. In addition, the review was performed by three co-authors, two of whom are among the original creators of the Galar dataset; this independence concern is not addressed.
- [Section III-E] The core assumption that the highest-mean GMM component corresponds exactly to mislabeled samples is asserted rather than tested. The third component is described as containing 'the highest loss values with strong outliers,' but high loss can equally indicate hard or ambiguous examples. Table I's large number of filtered clean samples is consistent with this alternative interpretation. The downstream F1 improvement may therefore result from pruning difficult examples rather than correcting labels. A comparison against simple uncertainty-based pruning (e.g., removing the same number of highest-loss samples without any GMM) is needed to support the mechanism claimed.
minor comments (5)
- [Section II] In the Related Work, 'achieve a F1-score5%for polyp detection' and '14%for blood detection' are missing spaces; please fix formatting.
- [Section III-D] The sentence 'which combine the suitability for low-power embedded devices' has a subject-verb agreement issue ('which combines' or 'MobileNetV3 ... combines').
- [Table II] The table header is difficult to parse: 'Uncleaned [32] Uncleaned' and similar entries would be clearer if the model source and cleaned/uncleaned status of each split were in separate columns.
- [Sections IV and V] The cleaned dataset splits are described as 'available here' but no URL or repository identifier is provided. A data-availability statement with a stable link is needed.
- [Section III-C] The term 'introduced primarily' is vague. Please specify exactly what fraction of the injected noise is drawn from each uncertainty quantile; otherwise the controlled experiment is not precisely reproducible.
Circularity Check
Two evaluation steps reduce to the pipeline's own outputs: biased noise injection into the controlled experiment and a self-filtered test-set row; the central cleaned-dev-set comparison remains non-circular but is confounded.
specific steps
-
self definitional
[Section III-C (Noise Injection, Kvasir-Capsule dataset); evaluated in Table I]
"For each sample, the average prediction confidence and entropy were computed across all epochs and three independent training runs, normalized, equally weighted, and combined into a single uncertainty score reflecting classification difficulty. Based on this score, samples were assigned to low-, mid-, and high-uncertainty quantiles. To imitate noisy samples, label noise was then introduced primarily by randomly selecting samples from the mid- and high-uncertainty groups."
In the controlled experiment, the ground-truth 'noisy' set is deliberately sampled from the high-uncertainty tail of the same network predictions that the GMM later uses (average per-epoch loss, confidence, entropy). High uncertainty and high loss are the same signal, so the GMM's highest-mean component will preferentially match the injected set even for clean but hard examples. Thus Table I's 456/471 recall is partly built into the sampling scheme; the 916 clean samples removed at 1% noise show the detector flags high-loss samples generally, not label errors specifically. This does not establish general mislabel detection.
-
fitted input called prediction
[Section IV, paragraph before Table II; Table II row 'Filtered Filtered']
"However, in the final run, the test set containing the corrected samples was additionally filtered to investigate if this leads to an additional classification enhancement. ... Filtered Filtered 91.72 73.67 89.88 68.05"
The test set is filtered using the same pipeline's noise-probability predictions before computing anomaly-detection metrics. Samples the method flags as mislabeled are removed from the evaluation set, so precision (89.88%) and F1 (73.67%) are measured on a subset chosen by the method itself. This guarantees an apparent improvement by removing the hardest/ambiguous test samples and cannot serve as independent evidence that those removed samples were actually mislabeled.
full rationale
The central cleaned-dev-set result (71.58 F1 on untouched test) is not definitionally circular: the dev-set labels are produced by the GMM pipeline while the test set is unmodified, so this is an internal comparison rather than a tautology—though it is confounded by possible hard-example pruning. However, two reported evaluation components do reduce to the pipeline's own outputs. First, the controlled Kvasir-Capsule experiment defines the 'mislabeled' set by drawing preferentially from mid/high-uncertainty scores derived from the same network whose average loss is then used by the GMM, so recall on that set is in part an artifact of the sampling scheme. Second, the 'Filtered Filtered' row filters the test set with the same algorithm being evaluated, so its high precision is by construction. The clinical validation is also weakened as an independent check because two of the three reviewing gastroenterologists are original Galar dataset creators and co-authors of this paper; this is a provenance concern rather than a derivation-level circularity. The cited baselines [31] and [32] are the authors' own prior work, but the main uncleaned-vs-filtered improvement is demonstrated internally and does not depend on those self-citations for its numeric reduction. Overall, the paper has substantial independent content, but the specific claim that the framework 'successfully detects incorrectly labeled data' is only partially supported by non-circular evidence, justifying a score of 6.
Axiom & Free-Parameter Ledger
free parameters (4)
- k_c (number of labels corrected) =
Not disclosed (Galar corrections: 31,650)
- k_f (number of labels filtered) =
Not disclosed (Galar filtered: 167,709; Kvasir false positives: 916/975/991)
- Number of GMM components =
3
- Uncertainty score weights =
0.5 confidence + 0.5 entropy
axioms (5)
- domain assumption Per-sample CNN training loss on noisy labels follows a three-component Gaussian mixture; the highest-mean component corresponds to mislabeled samples.
- domain assumption High loss is caused by label noise rather than by hard or ambiguous examples.
- domain assumption All meaningful label noise in these binary VCE tasks is a flip between normal and anomaly.
- domain assumption The Galar dataset splits of [31], [32] are appropriate and the uncleaned labels are the correct comparison baseline.
- domain assumption A majority vote of the three gastroenterologist co-authors is a reliable ground truth for mislabels.
read the original abstract
The classification performance of deep neural networks relies strongly on access to large, accurately annotated datasets. In medical imaging, however, obtaining such datasets is particularly challenging since annotations must be provided by specialized physicians, which severely limits the pool of annotators. Furthermore, class boundaries can often be ambiguous or difficult to define which further complicates machine learning-based classification. In this paper, we want to address this problem and introduce a framework for mislabel detection in medical datasets. This is validated on the two largest, publicly available datasets for Video Capsule Endoscopy, an important imaging procedure for examining the gastrointestinal tract based on a video stream of lowresolution images. In addition, potentially mislabeled samples identified by our pipeline were reviewed and re-annotated by three experienced gastroenterologists. Our results show that the proposed framework successfully detects incorrectly labeled data and results in an improved anomaly detection performance after cleaning the datasets compared to current baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
In: 2020 International conference on system science and engineering (ICSSE)
Chiu, Y .C., Tsai, C.Y ., Ruan, M.D., Shen, G.Y ., Lee, T.T.: Mobilenet- ssdv2: An improved object detection model for embedded systems. In: 2020 International conference on system science and engineering (ICSSE). pp. 1–5. IEEE (2020)
2020
-
[2]
In: 2022 25th Euromicro Conference on Digital System Design (DSD)
Christoph, G., Adrian, F., Tobias, H., Bernardo, P.P., L ¨ubeck, K., Oliver, B.: Hardware accelerator and neural network co-optimization for ultra-low-power audio processing devices. In: 2022 25th Euromicro Conference on Digital System Design (DSD). pp. 365–369. IEEE (2022)
2022
-
[3]
Gastroenterology123(4), 999–1005 (2002)
Costamagna, G., Shah, S.K., Riccioni, M.E., Foschia, F., Mutignani, M., Perri, V ., Vecchioli, A., Brizi, M.G., Picciocchi, A., Marano, P.: A prospective trial comparing small bowel radiographs and video capsule endoscopy for suspected small bowel disease. Gastroenterology123(4), 999–1005 (2002)
2002
-
[4]
Journal of the royal statistical society: series B (methodological)39(1), 1–22 (1977)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological)39(1), 1–22 (1977)
1977
-
[5]
Sensors20(23), 6718 (2020)
Feng, W., Quan, Y ., Dauphin, G.: Label noise cleaning with an adaptive ensemble method based on noise detection metric. Sensors20(23), 6718 (2020)
2020
-
[6]
In: Proceedings of the IEEE/CVF international conference on computer vision
Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y ., Pang, R., Vasudevan, V ., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1314–1324 (2019)
2019
-
[7]
arXiv preprint arXiv:1704.04861 (2017)
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Pith/arXiv arXiv 2017
-
[8]
Nature405(6785), 417–417 (2000)
Iddan, G., Meron, G., Glukhovsky, A., Swain, P.: Wireless capsule endoscopy. Nature405(6785), 417–417 (2000)
2000
-
[9]
Jiang, G., Zhang, J., Bai, X., Wang, W., Meng, D.: Which is more effective in label noise cleaning, correction or filtering? In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 12866– 12873 (2024)
2024
-
[10]
IEEE Transactions on Geoscience and Remote Sensing 56(10), 5673–5686 (2018)
Kang, X., Duan, P., Xiang, X., Li, S., Benediktsson, J.A.: Detection and correction of mislabeled training samples for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing 56(10), 5673–5686 (2018)
2018
-
[11]
Scientific Data12(1), 828 (2025)
Le Floch, M., Wolf, F., McIntyre, L., Weinert, C., Palm, A., V olk, K., Herzog, P., Kirk, S.H., Steinh ¨auser, J.L., Stopp, C., et al.: Galar-a large multi-label video capsule endoscopy dataset. Scientific Data12(1), 828 (2025)
2025
-
[12]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Lee, K.H., He, X., Zhang, L., Yang, L.: Cleannet: Transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5447–5456 (2018)
2018
-
[13]
arXiv preprint arXiv:2002.07394 (2020)
Li, J., Socher, R., Hoi, S.C.: Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394 (2020)
Pith/arXiv arXiv 2002
-
[14]
In: Proceedings of the IEEE international conference on computer vision
Lin, T.Y ., Goyal, P., Girshick, R., He, K., Doll´ar, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
2017
-
[15]
arXiv preprint arXiv:1711.05101 (2017)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Pith/arXiv arXiv 2017
-
[16]
John Wiley & Sons (2000)
McLachlan, G.J., Peel, D.: Finite mixture models. John Wiley & Sons (2000)
2000
-
[17]
https://www.medtronic.com/ covidien/en-nz/products/capsule-endoscopy/pillcam-sb-3-system.html/ (2025), [Online; accessed 7-May-2025]
Medtronic: PillCam™ SB3 System. https://www.medtronic.com/ covidien/en-nz/products/capsule-endoscopy/pillcam-sb-3-system.html/ (2025), [Online; accessed 7-May-2025]
2025
-
[18]
Monteiro, S., de Castro, F.D., Carvalho, P.B., Moreira, M.J., Rosa, B., Cotter, J.: Pillcam® sb3 capsule: Does the increased frame rate eliminate the risk of missing lesions? World journal of gastroenterology22(10), 3066 (2016)
2016
-
[19]
arXiv preprint arXiv:1705.01936 (2017)
Northcutt, C.G., Wu, T., Chuang, I.L.: Learning with confident examples: Rank pruning for robust classification with noisy labels. arXiv preprint arXiv:1705.01936 (2017)
Pith/arXiv arXiv 2017
-
[20]
In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops
Ostyakov, P., Logacheva, E., Suvorov, R., Aliev, V ., Sterkin, G., Khomenko, O., Nikolenko, S.I.: Label denoising with large ensembles of heterogeneous neural networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0–0 (2018)
2018
-
[21]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., Qu, L.: Making deep neural networks robust to label noise: A loss correction approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1944–1952 (2017)
1944
-
[22]
Journal of Machine Learning Research12, 2825–2830 (2011)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V ., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V ., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research12, 2825–2830 (2011)
2011
-
[23]
Pattern recognition39(4), 695–706 (2006)
Permuter, H., Francos, J., Jermyn, I.: A study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern recognition39(4), 695–706 (2006)
2006
-
[24]
arXiv preprint arXiv:1412.6596 (2014)
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., Rabinovich, A.: Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596 (2014)
Pith/arXiv arXiv 2014
-
[25]
In: European conference on computer vision
Sharma, K., Donmez, P., Luo, E., Liu, Y ., Yalniz, I.Z.: Noiserank: Un- supervised label noise reduction with dependence models. In: European conference on computer vision. pp. 737–753. Springer (2020)
2020
-
[26]
Scientific Data8(1), 142 (2021)
Smedsrud, P.H., Thambawita, V ., Hicks, S.A., Gjestang, H., Nedrejord, O.O., Næss, E., Borgli, H., Jha, D., Berstad, T.J.D., Eskeland, S.L., et al.: Kvasir-capsule, a video capsule endoscopy dataset. Scientific Data8(1), 142 (2021)
2021
-
[27]
In: Biomonitoring and Endoscopy Technologies
Swain, P., Iddan, G.J., Meron, G., Glukhovsky, A.: Wireless capsule endoscopy of the small bowel: development, testing, and first human trials. In: Biomonitoring and Endoscopy Technologies. vol. 4158, pp. 19–23. SPIE (2001)
2001
-
[28]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
Tanaka, D., Ikami, D., Yamasaki, T., Aizawa, K.: Joint optimization framework for learning with noisy labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
2018
-
[29]
Digestive diseases and sciences46, 2555–2566 (2001)
Thomson, A., Keelan, M., Thiesen, A., Clandinin, M., Ropeleski, M., Wild, G.: Small bowel review: diseases of the small intestine. Digestive diseases and sciences46, 2555–2566 (2001)
2001
-
[30]
In: Proceedings of the IEEE/CVF international conference on computer vision
Wang, Y ., Ma, X., Chen, Z., Luo, Y ., Yi, J., Bailey, J.: Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 322–330 (2019)
2019
-
[31]
In: International Workshop on Applications of Medical AI
Werner, J., Bause, O., Oexle, J., Floch, M.L., Brinkmann, F., Hampe, J., Bringmann, O.: Seeing more with less: Video capsule endoscopy with multi-task learning. In: International Workshop on Applications of Medical AI. pp. 12–21. Springer (2025)
2025
-
[32]
In: 2025 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
Werner, J., Gerum, C., Nick, J., Le Floch, M., Brinkmann, F., Hampe, J., Bringmann, O.: Enhanced anomaly detection for capsule endoscopy using ensemble learning strategies. In: 2025 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 1–7. IEEE (2025)
2025
-
[33]
In: International Workshop on Machine Learning in Medical Imaging
Werner, J., Gerum, C., Reiber, M., Nick, J., Bringmann, O.: Precise localization within the gi tract by combining classification of cnns and time-series analysis of hmms. In: International Workshop on Machine Learning in Medical Imaging. pp. 174–183. Springer (2023)
2023
-
[34]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Xiao, T., Xia, T., Yang, Y ., Huang, C., Wang, X.: Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2691–2699 (2015)
2015
-
[35]
Advances in neural information processing systems31(2018)
Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems31(2018)
2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.