{"id":"bb48feae-45a7-4515-95a3-5d51c68309a3","arxiv_id":"2602.06938","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A loss-based cleaning pipeline flags and corrects mislabeled frames in capsule endoscopy datasets, improving downstream anomaly detection, but the validation is biased.","lead":"This paper tests a pipeline that flags potentially mislabeled frames in two large capsule endoscopy datasets by looking at how hard the images are for a neural network to learn, then corrects or deletes the flagged frames. The authors report better disease-detection scores after cleaning, but the validation has several serious biases that make the headline claim shaky.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Controlled noise-injection is biased to high-uncertainty samples (III-C), so Table I does not show general mislabel detection; at 1% noise, 916 clean samples are removed vs 471 noisy, indicating the pipeline primarily identifies hard examples.","rationale":"The reader's weakest assumption, that high loss reflects hard examples rather than label errors, is exactly the load-bearing issue. My independent reading of Section III-C shows the noise injection is biased toward the same high-uncertainty samples the detector keys on, so the controlled experiment cannot distinguish these alternatives. The large numbers of clean samples filtered in Table I reinforce the concern, and the clinical validation only samples the very top of the score distribution, not the actual filtering population. The proposed concrete test would directly assess whether the GMM detects mislabels under a more realistic noise distribution. Since the current evidence does not support the central claim, the reader's REJECT verdict remains appropriate.","tokens_in":8813,"tokens_out":4347,"duration_ms":47829,"concrete_test":"Replicate the Kvasir controlled experiment with noise injected uniformly at random across all uncertainty quantiles (and separately only within the low-uncertainty quantile), keeping the same GMM pipeline and k_c/k_f settings. Report detection precision and recall for each noise level. If recall drops or precision falls further when noisy labels are not concentrated in high-uncertainty samples, the controlled experiment does not validate general mislabel detection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C states that label noise was 'introduced primarily by randomly selecting samples from the mid- and high-uncertainty groups.' Consequently, the injected noisy samples are exactly those whose loss the GMM is expected to inflate. The high recall in Table I (456/471 at 1%) is an artifact of this correlated sampling and does not establish that the GMM separates mislabels from difficult-but-correct examples. The false-positive counts are large: at 1% noise, 916 clean samples are filtered (vs 471 noisy), a precision of about 33% among filtered samples. At 5%, 975 clean samples are removed. The Galar precision estimate is based only on the top 100 of 500 highest-scoring samples, not on the 167,709 samples actually filtered from the dataset; the low precision in Table I implies the cleaned dev set contains many valid hard samples. Hence the Table II F1 gain (53.70→71.58) may be caused by pruning hard examples rather than correcting labels, so the central claim that the framework detects mislabeled data is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a mislabel-detection pipeline for Video Capsule Endoscopy (VCE) datasets. The pipeline trains a MobileNetV3 classifier, fits a three-component Gaussian Mixture Model (GMM) to per-sample training losses, corrects the k_c samples with the largest noise-reduction score, retrains, and then filters the k_f samples with the highest noise probability. It is evaluated in two stages: (1) a controlled noise-injection experiment on Kvasir-Capsule, and (2) application to the Galar dataset, followed by clinical review of 100 flagged samples by three gastroenterologist co-authors and retraining of an anomaly detector. The authors report F1-score improvements on Galar from 53.70 (uncleaned) to 71.58 (filtered dev set) and to 73.67 when the test set is also filtered, and claim that the framework reliably detects mislabeled data.","tokens_in":9102,"tokens_out":5402,"duration_ms":58545,"significance":"If the claims were sound, the work would be valuable for medical dataset curation: it addresses a real bottleneck in VCE and uses two large public datasets. The resulting cleaned splits could be reused by the community. However, the central claim that the framework 'successfully detects incorrectly labeled data' is not established by the current experiments. The controlled noise-injection protocol is biased toward high-uncertainty samples, the reported false-positive counts are very large, the 'Filtered Filtered' result is circular, and the clinical validation is small and not independent. The paper also leaves the key hyperparameters k_c and k_f unspecified. These are load-bearing issues, not presentation issues.","major_comments":[{"comment":"The controlled experiment does not test general mislabel detection because the injected noise is not random: label noise was 'introduced primarily by randomly selecting samples from the mid- and high-uncertainty groups.' Since the GMM in Section III-E flags samples by high loss, this selection guarantees that injected noisy samples are already concentrated in the high-loss region. The high recall in Table I (456/471 at 1%) is therefore an artifact of the injection protocol. More importantly, Table I shows 916 clean samples filtered at 1% injected noise, i.e., only about 33% of filtered/corrected samples are genuinely noisy. The pipeline is primarily identifying hard or uncertain examples, not label errors.","section":"Section III-C and Table I"},{"comment":"The 'Filtered Filtered' row evaluates the anomaly detector on a test set that was itself filtered by the same pipeline. The text states that the test set was 'additionally filtered to investigate if this leads to an additional classification enhancement.' This is circular: filtering the test set removes hard or uncertain examples and can inflate precision and F1-score. The row is not comparable to the baselines or to the 'Filtered Uncleaned' row, and the 89.88% precision in that row should not be used to support the claim of improved detection after cleaning.","section":"Table II, Section IV"},{"comment":"The number of corrected labels k_c and the number of filtered labels k_f are free parameters of the pipeline, but the manuscript never specifies how they are selected. These parameters directly determine the counts in Table I and the composition of the cleaned Galar dev set. Without a stated selection rule (e.g., based on a validation set or a fixed criterion independent of ground truth), the controlled experiment is not reproducible, and the reported cleaning results could be tuned. This is a load-bearing methodological gap.","section":"Sections III-B and IV"},{"comment":"The clinical validation is not representative of the pipeline's actual filtering behavior. The 100 reviewed samples are selected from the top 500 by noise-reduction score, whereas the pipeline filters 167,709 samples from Galar. Precision@100=78% on this hand-picked subset does not measure precision on the filtered set, and Table I shows that filtered sets contain many clean samples. In addition, the review was performed by three co-authors, two of whom are among the original creators of the Galar dataset; this independence concern is not addressed.","section":"Section III-F and Clinical Validation"},{"comment":"The core assumption that the highest-mean GMM component corresponds exactly to mislabeled samples is asserted rather than tested. The third component is described as containing 'the highest loss values with strong outliers,' but high loss can equally indicate hard or ambiguous examples. Table I's large number of filtered clean samples is consistent with this alternative interpretation. The downstream F1 improvement may therefore result from pruning difficult examples rather than correcting labels. A comparison against simple uncertainty-based pruning (e.g., removing the same number of highest-loss samples without any GMM) is needed to support the mechanism claimed.","section":"Section III-E"}],"minor_comments":[{"comment":"In the Related Work, 'achieve a F1-score5%for polyp detection' and '14%for blood detection' are missing spaces; please fix formatting.","section":"Section II"},{"comment":"The sentence 'which combine the suitability for low-power embedded devices' has a subject-verb agreement issue ('which combines' or 'MobileNetV3 ... combines').","section":"Section III-D"},{"comment":"The table header is difficult to parse: 'Uncleaned [32] Uncleaned' and similar entries would be clearer if the model source and cleaned/uncleaned status of each split were in separate columns.","section":"Table II"},{"comment":"The cleaned dataset splits are described as 'available here' but no URL or repository identifier is provided. A data-availability statement with a stable link is needed.","section":"Sections IV and V"},{"comment":"The term 'introduced primarily' is vague. Please specify exactly what fraction of the injected noise is drawn from each uncertainty quantile; otherwise the controlled experiment is not precisely reproducible.","section":"Section III-C"}],"recommendation":"reject","confidential_remarks":"The paper addresses a worthwhile problem and uses valuable public VCE datasets, but the current evidence does not support the central claim of reliable mislabel detection. The biased noise-injection protocol, circular test-set filtering, large false-positive counts, unspecified k_c/k_f, and non-independent clinical validation together require a substantially reworked evaluation. I would not oppose a future submission that reruns the controlled experiment with unbiased noise injection, evaluates solely on an untouched test set, states and justifies k_c/k_f, and provides an independent clinical assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The cleaned Galar splits are a real, citable resource, but the paper's central claim—that the pipeline reliably detects mislabeled frames—rests on a biased experiment and a selective clinical check.\n\nWhat's new: this is the first application of loss-GMM cleaning to the two largest public VCE datasets, it ships corrected/filtered splits, and it includes a Precision@100 evaluation with three gastroenterologists. The anomaly-detection jump after filtering (F1 71.6 vs 54.4) is eye-catching and likely useful to the community.\n\nThe evidence is weaker than the abstract suggests. The controlled noise injection in Section III-C preferentially samples from mid- and high-uncertainty groups—exactly the samples the GMM keys on—so Table I's recall is not a fair test. More damning, at 1% noise the pipeline filters 916 clean frames versus 471 noisy ones: precision around a third. That means it is mostly identifying hard examples, not mislabels. The clinical validation covers only the top 100 of 500 candidates, a fraction of the 167,709 frames actually filtered, and two of the three judges were involved in creating the Galar dataset. The 78% Precision@100 says little about the bulk. The Table II gain may therefore come from pruning ambiguous examples rather than correcting wrong labels; the \"Filtered Filtered\" row, which cleans the test set, is not comparable to the baselines.\n\nThat said, the artifact has value. A cleaned version of Galar, even if aggressively filtered, could be useful if the splits are shared. The method itself is standard loss-GMM, but the dataset is new and the paper is readable. The authors are transparent about the filtering volume.\n\nThis deserves a serious referee, but major revision is needed. The authors should run noise injection across all uncertainty strata, report precision/recall at filtering thresholds, validate a random sample of the actually filtered frames, and make the clinical panel independent or at least blinded. The cleaned dataset alone might justify a short paper even if the detection claims are tempered.\n\nFor peer review: send it out, with clear requests for these controls.","headline":"Useful artifact, shaky evidence: the cleaned VCE splits may help the community, but the paper's core claim about detecting mislabels is undercut by biased noise injection and high false positives.","tokens_in":9585,"tokens_out":3183,"would_cite":true,"duration_ms":31158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The central claim is that label noise, not model capacity, limits VCE anomaly detection, and a loss-mixture cleaning pipeline lifts F1 from 54% to 72%.","keywords":["video capsule endoscopy","mislabel detection","dataset cleaning","Gaussian mixture model","anomaly detection","label noise","medical imaging"],"falsifier":"An experiment with a dataset whose labels are all correct but whose samples vary widely in difficulty (e.g., by adding synthetic blur or occlusion to a fraction of clean images) would falsify the core assumption if the GMM's highest-mean component flags those clean-but-hard images as mislabels and the pipeline removes them. Concretely, compare the pipeline's false-positive removal rate on such a difficulty-only dataset against its performance on the same dataset with real injected label noise.","tokens_in":8726,"feed_emoji":"🩺","tokens_out":7200,"duration_ms":60016,"temperature":0.7,"pith_summary":"Video capsule endoscopy (VCE) datasets are large, class-imbalanced, and annotated by a small pool of physicians, so label noise is inevitable. This paper argues that a substantial portion of the difficulty in VCE anomaly detection is caused by mislabeled training frames rather than model architecture, and proposes an unsupervised cleaning pipeline to remove them. The pipeline trains a compact classifier, fits a three-component Gaussian mixture model to per-sample training loss, and treats the highest-loss component as mislabeled. After correcting and filtering those samples in the Galar dataset, anomaly detection F1 rises from 54.38% (previous best) to 71.58%, with more confident predictions. Clinical review of the 100 most-suspected frames confirmed 78% as genuinely mislabeled, supporting the pipeline's reliability.","feed_headline":"One cleaning pass lifts capsule-endoscopy F1 from 54% to 72%","feed_subtitle":"A loss-mixture pipeline removes mislabeled frames from Galar and improves detector accuracy and confidence.","key_machinery":"The key mechanism is a three-component Gaussian Mixture Model (GMM) fitted to the per-sample average training loss of a compact convolutional classifier. The component with the highest mean is interpreted as the mislabeled population; a noise-reduction score (the change in noise probability after a label correction) guides which samples to flip, and a final step filters the highest-noise samples. This transforms the model's own loss signal into a ranking of suspected label errors.","core_discovery":"The central discovery is that per-sample training loss separates clean, hard, and mislabeled samples well enough to drive a full dataset-cleaning workflow. On Kvasir-Capsule with controlled label flipping, the pipeline recovers 456/471 injected mislabels at 1% noise and 2262/2360 at 5%; on the Galar dataset it filters 167,709 samples (4.8%) and corrects 31,650 (0.9%). Training an anomaly detector on the cleaned development set yields 93.83% accuracy and 71.58% F1 on the untouched test set, compared with 54.38% for the best earlier baseline and 37.01% for another. The authors further show that cleaning increases the model's maximum confidence on test predictions (0.85 to 0.96), and that three","pith_inferences":["The pipeline's own numbers show a high false-positive rate on clean data: at 1% injected noise, 916 of 1,372 removed samples were actually clean. This suggests part of the F1 gain may come from pruning hard, ambiguous examples rather than from correcting label errors; a controlled comparison against random or uncertainty-based pruning would separate those effects.","The GMM's three-component assumption couples label noise with sample difficulty. In datasets where the hardest clean examples are as rare as mislabels, the highest-mean component may mix both populations, so the method's performance likely degrades gracefully only if true noise is the dominant source of high loss.","If the cleaned Galar splits become widely used, baseline comparisons in subsequent VCE anomaly-detection papers will need to state explicitly whether they train on the cleaned or original development set, since the 17-point F1 gap makes the choice decisive.","A testable extension: running the same pipeline with different classifier capacities and loss functions should shift the GMM separation; measuring how the precision of mislabel detection changes across those shifts would map the method's reliability boundary."],"forward_implications":["Cleaning the training set before anomaly detection improves F1 on the Galar benchmark by 17 percentage points over the best prior baseline, without changing the test set or the model architecture.","Models trained on cleaned data produce more confident predictions (maximum confidence 0.85 to 0.96), which is relevant for on-device screening decisions.","The released corrected and filtered dataset splits give the community a ready-to-use cleaned training set for VCE anomaly detection.","Because the pipeline is unsupervised and requires only a trained classifier and its loss values, it can be applied to other medical datasets with ambiguous class boundaries."],"fun_headline_variants":["Mislabel detection lifts capsule-endoscopy F1 from 54% to 72%","Loss-based mislabel filtering improves VCE anomaly detection","Cleaning mislabeled frames raises capsule-endoscopy F1 by 18 points","VCE data cleaning recovers 456/471 injected mislabels","Mislabel pipeline improves capsule-endoscopy F1 to 71.58%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pipeline assumes that the training examples a model finds hardest to learn, measured by the largest losses, are the ones whose labels are wrong, rather than simply being difficult or ambiguous images.","fun_headline_variants_meta":{"raw":{"variants":["Mislabel detection lifts capsule-endoscopy F1 from 54% to 72%","Loss-based mislabel filtering improves VCE anomaly detection","Cleaning mislabeled frames raises capsule-endoscopy F1 by 18 points","VCE data cleaning recovers 456/471 injected mislabels","Mislabel pipeline improves capsule-endoscopy F1 to 71.58%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3174,"prompt_tokens":716,"completion_tokens":2458,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":2362}},"tokens_in":460,"tokens_out":2458,"duration_ms":15876,"temperature":1.0,"reasoning_tokens":2362,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:41:32.054402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment with a dataset whose labels are all correct but whose samples vary widely in difficulty (e.g., by adding synthetic blur or occlusion to a fraction of clean images) would falsify the core assumption if the GMM's highest-mean component flags those clean-but-hard images as mislabels and the pipeline removes them. Concretely, compare the pipeline's false-positive removal rate on such a difficulty-only dataset against its performance on the same dataset with real injected label noise.","supporting_citations":[],"review_version":1}