{"id":"7987c2ab-76b1-4dfc-a905-7f37c8b299c8","arxiv_id":"2412.02313","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new fine-grained benchmark dataset of ostracod images shows that existing robust-learning and label-correction methods do not outperform cross-entropy or a naive ensemble baseline.","lead":"The paper introduces Noisy Ostracods, a real-world dataset of 71,466 ostracod images with natural label noise and extreme class imbalance, and benchmarks robust learning and label-correction methods on it. The main finding is that standard cross-entropy training matches or beats sophisticated noise-robust methods, and a simple ensemble cross-validation baseline detects more label errors than specialized tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main conclusion is statistically under-supported: Appendix H means show PLM accuracy 96.70±0.06 vs CE 95.77±0.35 on ResNet-50 (Welch t≈5.9, p≈0.003), so the claim that no LNL method significantly improves over CE needs a paired test or a metric-specific qualifier.","rationale":"I read the paper as a dataset/benchmark contribution whose headline empirical finding is negative: robust methods fail to beat CE, and NECV beats specialized label-error detectors. The dataset, noise taxonomy, and NECV baseline are valuable and honestly documented, including the information-leakage caveats in Appendices E and H. I do not object to the dataset's release or to the NECV comparison per se. My main concern targets the robustness of the negative result. The appendix actually contains the seeds of a counterexample: PLM's accuracy is higher and, on ResNet-50, the difference looks statistically significant even from the summary statistics. This matters because the abstract/conclusion make a broad 'no significant improvement' claim, and the main table's best-of-5 selection can mask or create differences depending on seed. A paired significance test is cheap and would settle it. If the test confirms significance, the conclusion should be narrowed; if not, the current claim can stay. This is related to the reader's mention of best-of-5 reporting, though my emphasis is on the statistical conclusion rather than the ground-truth reliability concern; hence partial agreement.","tokens_in":18753,"tokens_out":9570,"duration_ms":110217,"concrete_test":"Run a paired permutation test (or Welch t-test on the raw per-run numbers) comparing each method against CE for accuracy and macro-F1 on both backbones using the five runs behind Appendix H Tables 4 and 5, available in the GitHub repository. If PLM's accuracy advantage is significant (p<0.05) on either backbone, revise the abstract/conclusion to 'no significant improvement in macro-F1/precision/recall' or report the accuracy caveat explicitly; if not significant across all metrics, the current claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative result is that robust LNL methods 'have not yielded significant performance improvements' over CE. The main Table 1 reports only the best of 5 runs. Appendix H provides the 5-run means: for ResNet-50, PLM accuracy is 96.70±0.06 vs CE 95.77±0.35 (Table 4). An approximate Welch t-test on these summaries gives t≈5.9, p≈0.003; for ViT-B-16, PLM is 96.10±0.37 vs CE 94.21±1.59 (Table 5), t≈2.6, p≈0.055. Thus at least on the primary backbone and accuracy metric, PLM appears significantly better than CE, directly contradicting a blanket reading of the abstract. The text acknowledges PLM's higher accuracy but dismisses it as 'around 1%' with no significance test, and the conclusion is phrased in terms of no significant advantage. The claim may survive if the intended headline metric is macro-F1, precision, or recall (where CE remains numerically ahead), or if 'significant' is meant as practically meaningful, but the paper does not say this. Without paired tests across metrics, the central claim is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Noisy Ostracods, a real-world fine-grained image dataset for ostracod genus and species classification with an estimated 5.58% genus-level noise ratio, extreme class imbalance, and multiple real-world noise types including feature errors, pseudo-classes, mixed classes, and open-set classes. The authors manually cleaned the 14,320 test and validation samples and report benchmark results comparing cross-entropy training with several Learning with Noisy Labels methods on ResNet-50 and ViT-B-16. They also compare label-error detection methods against a simple ensemble cross-validation baseline (NECV). The main claims are that LNL methods do not provide significant improvements over cross-entropy training on this dataset and that NECV detects label errors more effectively than specialized methods such as Confident Learning and SimiFeat.","tokens_in":19059,"tokens_out":5028,"duration_ms":48212,"significance":"If the claims hold, the dataset is a valuable addition to the sparse collection of real-world noisy benchmarks, particularly for fine-grained taxonomy and extreme class imbalance. The authors provide a detailed characterization of real-world noise sources, a manually curated evaluation set, transition matrices, and reproducible evaluation protocols, with code and data released. The paper is also explicit about limitations, including the lack of a cleaned training set. The NECV baseline is simple and transparent, and the negative results for specialized LNL methods are informative for the community. However, the key negative claim about LNL methods requires more rigorous statistical support before the benchmark's headline conclusion can be accepted.","major_comments":[{"comment":"The central claim that LNL methods 'have not yielded significant performance improvements' over CE is not supported by the reported statistics. Table 1 reports only the best of 5 runs, while the Appendix H means show PLM with ResNet-50 at 96.70±0.06 vs CE at 95.77±0.35, which a Welch t-test on the summary statistics would reject as equal (approximate t≈5.9, p≈0.003); for ViT-B-16 the corresponding means are 96.10±0.37 vs 94.21±1.59 (approximate p≈0.055). The text acknowledges PLM's higher accuracy but dismisses it as 'around 1%' without a significance test. The claim may survive if restricted to macro-F1, precision, or recall, or if 'significant' is defined as practically meaningful, but the paper must either report paired tests across metrics or explicitly qualify the claim. As written, the abstract and conclusion overstate the negative result.","section":"§4.1 and Appendix H, Tables 4–5"},{"comment":"The 5.58% noise ratio is stated in the abstract as an estimate for the whole dataset, but it is measured only on the 14,320 manually cleaned test/validation samples, while the training set remains uncleaned. The paper itself lists the absence of a cleaned training set as the biggest limitation. Since the training set constitutes 80% of the data and may have a different noise distribution (e.g., different projects and annotation years, as shown in Table 3), the extrapolation from the cleaned 20% to the full 71,466 samples is an unsupported assumption. Please rephrase the abstract and Section 3.3 to state explicitly that 5.58% is the confirmed noise ratio in the cleaned test/validation portion, and discuss the uncertainty in extrapolating it to the training set.","section":"§3.3 and §5"},{"comment":"The main table's choice of reporting the best of 5 runs, rather than the mean, makes the headline comparison unreliable and is inconsistent with the variability shown in Appendix H. For example, Loss-clip on ResNet-50 has mean accuracy 62.43±32.98 across runs, so its Table 1 value of 95.79 is not representative. Please move the mean and variance (or a compatible confidence interval) into the main table, or add per-run results in the appendix with paired comparisons, so that the reader can evaluate the 'no significant improvement' claim directly from the main text.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"Please clarify that the benchmark experiments are genus-level classification; the abstract mentions both genus and species classification, but the cleaned evaluation sets and all reported results are at the genus level, and species-level cleaning is explicitly deferred to future work.","section":"Abstract and §3.3"},{"comment":"There are several typos: 'Expect the transition matrix whose result is not directly incomparable' should be 'Except for the transition matrix, whose result is not directly comparable', and 'aggrated' should be 'aggregated'.","section":"§4.1"},{"comment":"The table header formatting is difficult to parse ('Hit', 'Feature', 'Label', 'Fix', 'Found P.', 'Hit rate' are packed together). Please reformat the header so that each column has a clear name, and define 'Fix prec.' explicitly, since the text discusses precision only generically.","section":"§4.2, Table 2"},{"comment":"The notation in Equation (1) is confusing: sup[P*(yi∈m*|xi)] is written as if it were a supremum over a random variable; please rewrite the definition of feature error in plain language with clearer probabilistic notation.","section":"§3.2, Eq. (1)"},{"comment":"The transition-matrix experiment is described as an upper bound in the appendix, but Section 4.1 uses its poor performance to conclude that class-dependent noise assumptions fail. This is reasonable, but the main text should remind the reader that the low classification accuracy of this method is partly due to the 51-class subset and the training protocol, not solely to the invalidity of the class-dependent assumption.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline between minor and major revision. The dataset contribution is solid and the release of code and data is a strength, but the headline negative claim about LNL methods is the kind of load-bearing empirical statement that a benchmark paper should support with proper statistical tests. The authors' own appendix data appear to contradict a blanket reading of the abstract, so this needs to be fixed rather than merely clarified. I would also suggest that the authors consider paired tests across seeds for all metrics, since the between-run variance is high for several methods and the best-of-5 reporting in Table 1 masks this."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. The paper ships something genuinely new: 71,466 ostracod images, fine-grained genus/species labels, an imbalance factor of 22,429, and real noise types that don't show up in CIFAR-N or Clothing1M — pseudo-classes, mixed classes, an open-set new genus, and feature-level corruption (bad photos, fragments, position errors). The expert-cleaned 20% test/validation split, with transition matrices between original and corrected genus labels, is a legitimately useful evaluation base for label-correction methods. That alone earns a serious look.\n\nThe headline negative finding — robust LNL methods give no significant gain over plain cross-entropy — is the valuable part, but it is not established by the paper's own numbers. Appendix H gives 5-run means, and on accuracy PLM beats CE on both backbones: 96.70±0.06 vs 95.77±0.35 on ResNet-50, 96.10±0.37 vs 94.21±1.59 on ViT-B. An approximate test on those summaries gives p≈0.003 on ResNet-50. The authors acknowledge PLM's higher accuracy but wave it off as 'around 1%' with no significance test. The claim might survive if the intended metric is F1, precision, or recall — where CE does stay ahead — but the abstract doesn't qualify it that way. So the central conclusion as phrased is statistically under-supported.\n\nTwo reporting problems compound this. Main Table 1 reports the best of five runs, not the mean, and the appendix shows why that's misleading: Loss-clip's mean accuracy on ResNet-50 is 62.43±32.98 while the table shows 95.79. A benchmark table built on best runs can't carry claims about method ranking. Second, the transition-matrix and Dynamic Loss setups leak information (true transition matrix; cleaned validation as meta set) — the paper admits both, in Appendices E and H, and that honesty is real credit.\n\nWhat I'd also flag: the 5.58% noise rate is measured on the cleaned 20%; Section 5 admits the training set was never cleaned, so the observation that CE's 4.02% error is below the dataset's noise rate rests on an extrapolation. And the repository URL can't be verified from the manuscript itself. On the label-correction side, the main result — a simple 14-model ensemble cross-validation (NECV) finds 71% of errors vs Confident Learning's 59% — is credible and correctly credited to prior work.\n\nWho this is for: learning-with-noisy-labels researchers who need a real, hard, imbalanced, fine-grained noisy benchmark, and computational taxonomy people building identification systems. The dataset is the contribution, and the negative results are useful even if the abstract overstates them. I'd send it to peer review — a serious referee is warranted — with the fixes being: report means, run paired tests or qualify the claim by metric, and make the data actually reachable.","headline":"A genuinely new real-world noisy dataset whose abstract overstates the negative result: on accuracy, PLM beats CE by ~1% and the paper never runs a significance test.","tokens_in":19564,"tokens_out":7793,"would_cite":true,"duration_ms":83226,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that on a real fine-grained ostracod dataset, learning-with-noisy-labels methods provide no significant gain over standard cross-entropy training, and a naive ensemble baseline detects label errors better than Confident…","keywords":["noisy labels","fine-grained classification","real-world benchmark","label correction","class imbalance","ostracod taxonomy","learning with noisy labels","ensemble cross-validation"],"falsifier":"Have a second, independent panel of ostracod experts re-examine a random sample of the 14,320 'cleaned' images without seeing the original corrections; if the confirmed noise rate moves substantially, or if re-running the Table 2 methods against the new labels changes NECV's lead, the reported rankings are an artifact of that curation. Alternatively, clean the full training set and check whether cross-entropy's 4.02% error rate and the LNL method rankings persist on held-out data.","tokens_in":18587,"feed_emoji":"🦐","tokens_out":6003,"duration_ms":59543,"temperature":0.7,"pith_summary":"This paper introduces Noisy Ostracods, a 71,466-image benchmark of ostracod shells with expert-vetted labels, and asks two questions: do learning-with-noisy-label (LNL) methods train more accurate classifiers than standard cross-entropy, and do specialized label-correction methods find more mislabeled images than a simple ensemble baseline? On this dataset, the answer to both is no. Cross-entropy training with ImageNet-pretrained backbones matches or beats every tested robust-training method, and a naive ensemble of 14 models cross-validated over ten splits (NECV) detects 71.19% of the manually confirmed label errors, outperforming Confident Learning (59.37%) and all other correction methods tested. The authors argue that the dataset's fine-grained classes, extreme imbalance ($\\rho = 22429$), and noise types—including pseudo-classes, mixed classes, open-set new genera, and image-quality failures—make it a harder, more realistic test than synthetic-noise benchmarks, and they release it to push robust-learning research toward real-world conditions.","feed_headline":"14-model vote finds 71% of bad labels; specialized methods trail","feed_subtitle":"On the Noisy Ostracods benchmark, simple cross-validation ensembling beats Confident Learning's 59% hit rate.","key_machinery":"The load-bearing object is the Noisy Ostracods dataset itself, a fine-grained image benchmark with 78 annotated genera and 138 annotated species, an imbalance factor $\\rho = 22429$, a confirmed 5.58% genus-level noise rate on the 14,320 manually curated test/validation samples, and a documented taxonomy of noise types (feature errors such as photographic, fragmentation, position, and preprocessing problems; label errors such as pseudo-classes, mixed classes, new classes, and hard classes). The label-correction result is carried by the NECV algorithm: split data into 10 folds, train 14 diverse pretrained models on nine folds, count agreement with the original label across the 14 predictions, and flag agreement below 0.5. The noise transition matrix with the true, hand-derived transition probabilities is the mechanism used to test class-dependent noise assumptions.","core_discovery":"The paper's central discovery is that current noisy-label machinery does not deliver its advertised benefits on a real fine-grained taxonomy task. Across two backbones and eleven methods, standard cross-entropy training on raw noisy labels either outperforms or matches every LNL baseline; the only method to edge past CE in accuracy, PLM, does so by roughly one percentage point while sacrificing precision, recall, and F1. On the label-cleaning side, the bespoke detectors—Confident Learning, SimiFeat, AUM, and CINCER—all fall short of NECV, a baseline that simply counts how many of 14 diverse pretrained models agree with the original label and flags samples with less than half agreement. When the majority-wrong genus Paradoxostomid is excluded, NECV's hit rate rises above 90%. The authors interpret these results as evidence that assumptions underlying class-dependent noise models, such as the noise transition matrix, do not hold here, and that pre-trained models already absorb much of the real-world noise that specialized methods are designed to remove.","pith_inferences":["NECV's advantage probably reflects ensemble diversity and its ability to catch majority-wrong classes, but the paper does not ablate the number of models or the agreement threshold, so a cheaper protocol with three or four architectures might recover most of the hit rate.","Because the training set is uncurated, the robustness rankings could shift once its noise distribution is measured; the released benchmark may currently understate the difficulty of cleaning the full dataset.","The documented project-specific errors and pseudo-class conventions suggest that metadata such as project ID, annotator, and cf./aff. markers could be exploited as privileged information by future label-correction methods, an avenue the paper identifies but does not pursue.","If the NECV result transfers, museum-style taxonomic imaging projects could assemble cleaning pipelines from off-the-shelf pretrained models and cross-validation rather than specialized noise detectors, lowering the barrier to building clean reference datasets."],"forward_implications":["ImageNet-pretrained cross-entropy training becomes the natural reference point for any new LNL method aimed at fine-grained taxonomy data.","Real-world benchmarks like this one are needed because synthetic-noise results on CIFAR-10-style data do not predict which methods will help in practice.","Label-cleaning tools should be compared against ensemble cross-validation baselines before being adopted in dataset curation pipelines.","Methods built on class-dependent noise transition assumptions will need to be extended, since even the true transition matrix underperforms in this setting.","The released cleaned validation and test labels give future work a fixed reference for measuring both classification robustness and cleaning effectiveness."],"supporting_citations":[{"why":"Confident Learning, the main label-correction baseline whose 59.37% hit rate NECV outperforms.","marker":"[13]"},{"why":"SimiFeat, the feature-similarity label-noise detector that underperforms on this dataset.","marker":"[12]"},{"why":"CIFAR-10N, the real-world human-annotated benchmark that motivates the synthetic-to-real gap claim.","marker":"[20]"},{"why":"Co-teaching, one of the robust-training methods evaluated and outperformed by cross-entropy training.","marker":"[10]"},{"why":"DivideMix, a state-of-the-art LNL method whose poor performance illustrates the synthetic-to-real gap.","marker":"[34]"},{"why":"AQuA benchmark whose evaluation protocol and methods (AUM, CINCER) the paper adapts for label correction.","marker":"[56]"},{"why":"Prior ensemble noise-elimination idea that NECV formalizes into a strong baseline.","marker":"[45]"},{"why":"Endless Forams, source of the negative-class images added to the dataset.","marker":"[19]"}],"fun_headline_variants":["Simple vote beats fancy label-fixers on noisy ostracods","14-model vote finds 71% bad labels; fancy methods fail","For noisy taxonomy, a voting baseline outsmarts robust ML","Ostracod labels: 14-model vote wins, Confident Learning loses","On fine-grained noise, cross-entropy holds its own vs 11 methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The manual corrections of the 14,320 test and validation images are treated as error-free ground truth, and the 5.58% noise ratio measured there is assumed to represent the noise in the 57,000-image uncurated training set.","fun_headline_variants_meta":{"raw":{"variants":["Simple vote beats fancy label-fixers on noisy ostracods","14-model vote finds 71% bad labels; fancy methods fail","For noisy taxonomy, a voting baseline outsmarts robust ML","Ostracod labels: 14-model vote wins, Confident Learning loses","On fine-grained noise, cross-entropy holds its own vs 11 methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000921,"raw_usage":{"total_tokens":4030,"prompt_tokens":1106,"completion_tokens":2924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":2829}},"tokens_in":722,"tokens_out":2924,"duration_ms":21862,"temperature":1.0,"reasoning_tokens":2829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:35:30.322711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second, independent panel of ostracod experts re-examine a random sample of the 14,320 'cleaned' images without seeing the original corrections; if the confirmed noise rate moves substantially, or if re-running the Table 2 methods against the new labels changes NECV's lead, the reported rankings are an artifact of that curation. Alternatively, clean the full training set and check whether cross-entropy's 4.02% error rate and the LNL method rankings persist on held-out data.","supporting_citations":[{"cited_title":"Confident Learning: Estimating Uncer- tainty in Dataset Labels","cited_arxiv_id":null,"evidence_quote":"Confident Learning, the main label-correction baseline whose 59.37% hit rate NECV outperforms."},{"cited_title":"Detecting Corrupted Labels Without Training a Model to Predict","cited_arxiv_id":null,"evidence_quote":"SimiFeat, the feature-similarity label-noise detector that underperforms on this dataset."},{"cited_title":"Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations","cited_arxiv_id":null,"evidence_quote":"CIFAR-10N, the real-world human-annotated benchmark that motivates the synthetic-to-real gap claim."},{"cited_title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","cited_arxiv_id":null,"evidence_quote":"Co-teaching, one of the robust-training methods evaluated and outperformed by cross-entropy training."},{"cited_title":"DivideMix: Learning with Noisy Labels as Semi-supervised Learning","cited_arxiv_id":null,"evidence_quote":"DivideMix, a state-of-the-art LNL method whose poor performance illustrates the synthetic-to-real gap."},{"cited_title":"AQuA: A Benchmarking Tool for Label Quality Assessment","cited_arxiv_id":null,"evidence_quote":"AQuA benchmark whose evaluation protocol and methods (AUM, CINCER) the paper adapts for label correction."},{"cited_title":"Ensemble Methods for Noise Elimination in Classifi- cation Problems","cited_arxiv_id":null,"evidence_quote":"Prior ensemble noise-elimination idea that NECV formalizes into a strong baseline."},{"cited_title":"Endless Forams: >34,000 modern planktonic foraminiferal images for taxo- nomic training and automated species recognition using convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Endless Forams, source of the negative-class images added to the dataset."}],"review_version":1}