{"id":"0043698e-327c-4a54-b402-c76cf355d9b8","arxiv_id":"2412.02596","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reconstruction error ratios from class-wise autoencoders correlate with dataset classification difficulty and detect mislabeled images on hard datasets.","lead":"This paper introduces a cheap way to measure how hard a classification dataset is and to flag likely mislabeled images, by training a small autoencoder for each class and comparing how well each image is rebuilt by its own class versus other classes. The scores roughly track the accuracy of state-of-the-art models and beat other feature-based methods at finding label mistakes in difficult datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1-level SOTA mislabel claim rests on hand-tuned threshold ansatz (Eq. 9) and noise estimator (Eq. 8) that is biased on DeepWeeds; without held-out validation the claim is not yet supported.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the F1-level mislabel detection results depend on a threshold ansatz with hand-set constants and a noise estimator whose key approximation is only argued for small symmetric noise and is visibly biased on DeepWeeds. I agree with this assessment. The dataset-difficulty correlation (ρ=0.639, or 0.780 with Oxford Flowers excluded) is suggestive but depends on only 19 points and one excluded outlier, so it is not the most impressive claim. The headline claim in the abstract and Section 5 is SOTA mislabel detection, and that claim is conditional on the tuned threshold. The threshold-free AUROC results provide partial independent support, and the released code is a real asset, but they do not rescue the F1-level SOTA claim if the threshold constants do not transfer. The paper's own admission that the ansatz overshoots for confidence-based and human-annotator noise reinforces the regime-dependence. Therefore the conditional verdict is appropriate and no change is needed; the proposed held-out test would settle whether the concern actually lands.","tokens_in":24794,"tokens_out":2947,"duration_ms":30678,"concrete_test":"Hold out a set of datasets not used to set the constants (e.g., the 10 MedMNISTv2 datasets or a random 5 of the 19) and recompute the F1 curves of Fig. 6 using the published constants γ4=1.01, γ5=1.5, γ6=13.8, comparing against the per-dataset oracle threshold that maximizes F1. If the fixed-constant F1 falls more than 5 points below the oracle on average for symmetric/asymmetric noise, the SOTA F1 claim is overfit to the tuning set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mislabel-detection claim—SOTA F1 under symmetric/asymmetric noise—depends on the threshold ansatz of Eq. (9) with constants γ4=1.01, γ5=1.5, γ6=13.8, which the paper states are 'close to optimal' on the same benchmark datasets. The F1 curves in Fig. 6 therefore evaluate a tuned threshold, not a parameter-free score. The ansatz also requires η from Eq. (8), whose derivation (Appendix B.2) assumes the approximation Δ_˜c(x_c) ≈ min Δ(x_c), valid only for small symmetric noise; Fig. 9 shows a large bias on DeepWeeds (intercept 0.207). If the constants or the η estimate do not transfer to new datasets, the F1-level SOTA claim collapses, leaving only the threshold-free AUROC evidence (Figs. 13–14), which is more modest and limited to hard datasets. The paper itself notes the ansatz overshoots for confidence-based and human-annotator noise, so the claim is already regime-dependent. No internal inconsistency is alleged; the concern is external validity of the tuned threshold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reconstruction Error Ratios (RERs): for each class, a shallow autoencoder (reconstructor) is trained on foundation-model features, and the ratio chi(x_c) = Delta_c(x_c)/min_{c' != c} Delta_{c'}(x_c) is used as a sample-level difficulty score. The authors claim that the dataset-level average chi correlates with state-of-the-art classification error rate across 19 visual datasets, that chi_n follows a universal finite-size scaling law (Eq. 5) enabling a first-time decomposition of difficulty into finite-size and Bayes-error/boundary components, and that sample-level chi serves as a label-mistakenness score achieving state-of-the-art feature-based mislabel detection under symmetric and asymmetric noise. The paper reports both F1 results using a hand-set threshold ansatz (Eq. 9) and threshold-free AUROC results (Figs. 13-14), and includes experiments on 19 datasets, an out-of-domain medical benchmark, and robustness checks across feature backbones.","tokens_in":25182,"tokens_out":4529,"duration_ms":45359,"significance":"If the central claims hold, RERs would be a valuable practical tool: one cheap pass on frozen foundation-model features yields dataset-level difficulty and sample-level mislabel scores without training a classifier. The paper has clear strengths: the method is efficient and interpretable; the code is publicly released; the dataset-level difficulty claim is grounded in an external criterion (SOTA error rates); and the threshold-free AUROC comparisons provide evidence that the ranking signal is not entirely an artifact of the tuned threshold. The mislabel-detection results on hard datasets under symmetric and asymmetric noise are a useful contribution even if the F1-level SOTA claim is not yet fully supported. However, the current evidence for the headline F1 claim depends on constants that are tuned on the same benchmark datasets, and the correlation with SOTA error is moderate (rho = 0.639, R^2 about 0.41) rather than 'strong' as stated in the abstract.","major_comments":[{"comment":"The F1-level SOTA mislabel detection claim in Section 5 and Fig. 6 is not yet supported, because the threshold ansatz (9) uses constants gamma4=1.01, gamma5=1.5, gamma6=13.8 that the paper states are 'close to optimal' on the same benchmark datasets. The optimal threshold varies across datasets and noise types (Appendix C.1), so the F1 curves evaluate a tuned threshold rather than a parameter-free score; no held-out validation is reported. Please specify how these constants were selected, apply them to untouched datasets or noise configurations, or present the threshold-free AUROC results (Figs. 13-14) as the primary mislabel-detection evidence.","section":"§5, Eq. (9)"},{"comment":"The noise estimator eta about (chi0 - chirand)/(1 - chirand) used by the threshold ansatz relies on the approximation Delta_tilde_c(x_c) approximately min_c Delta_c(x_c) (Appendix B.2), which is justified only for small symmetric noise. Figure 9 shows substantial systematic bias even on the training datasets, e.g., DeepWeeds has intercept 0.207, and Eq. (9) depends directly on this eta. Please quantify the sensitivity of the F1 scores to the noise-estimation bias and test the eta estimator under asymmetric and confidence-based noise beyond the low-noise symmetric regime.","section":"§4.3, Appendix B.2, Eq. (8)"},{"comment":"The claimed decomposition of classification difficulty into finite-size and Bayes-error/boundary contributions rests on the fitted rational function (5) with a universal exponent gamma0=1.808 and per-dataset constants, and the extrapolated chi_infinity is then interpreted as the difficulty at infinite data. No independent validation of chi_infinity against a second estimator of Bayes error is provided, so the statement that this is the 'first time such a separation has been explicitly possible' (Section 4.2) goes beyond the evidence. Please add an out-of-sample check of the extrapolation, for example by fitting Eq. (5) on n <= n0 and comparing predictions to measured chi_n at n > n0, or by comparing chi_infinity to an independent estimate of irreducible error.","section":"§4.2, Eq. (5)"},{"comment":"The central difficulty claim is supported by a Pearson correlation of rho=0.639 (R^2 about 0.41) between log-error rate and chi across 19 datasets; after excluding Oxford 102 Flowers the correlation rises to rho=0.780, but the exclusion is post hoc. Calling this 'strong' in the Abstract and Conclusion overstates a moderate relationship, and the text's appeal to top-5 accuracy for large-class datasets (ImageNet, SUN397, Places205) is a plausible explanation rather than a correction. Please report the correlation with confidence intervals, include alternative rank correlations, and state the post hoc exclusion clearly in the Abstract.","section":"§4.1, Fig. 3"}],"minor_comments":[{"comment":"The threshold definition says 'maximizes our F1 score' but writes chi* = arg min_{chi_thresh} F1(chi_thresh); this should be arg max.","section":"Appendix C.1, Eq. (22)"},{"comment":"The definition of hard datasets is inconsistent: the text says 'easy (SOTA accuracy > 0.95) and hard (SOTA accuracy > 0.95)' while the surrounding discussion and Fig. 14 use SOTA accuracy < 0.95 for hard; please correct the typo.","section":"Appendix C.2, Fig. 14"},{"comment":"The sentence 'we compare our reconstruction error-based method to two three alternatives' contains a typo and should read 'to three alternatives'.","section":"§A.3.2"},{"comment":"The claim that RERs 'are defined similarly to' the silhouette-score-based difficulty scores of Scheidegger et al. is vague; a concrete comparison of the two formulas in one sentence would help the reader assess novelty.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's abstract somewhat overstates the evidence: the F1-level SOTA mislabel claim depends on constants tuned on the same benchmarks, and the dataset-difficulty correlation is moderate before a post hoc exclusion. The threshold-free AUROC results are more convincing and should perhaps be foregrounded. The finite-size decomposition claim would benefit from independent validation of the extrapolated limit. This is a fixable set of issues within the manuscript's scope, so major revision rather than rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful empirical paper: class-wise autoencoders on frozen CLIP/DINOv2 features produce a reconstruction-error-ratio score that tracks SOTA classification error across 19 datasets, and the threshold-free AUROC results on hard datasets under symmetric/asymmetric noise are reasonably convincing. Second, the abstract overstates the mislabel-detection contribution. The F1 curves in Fig. 6 come from a threshold ansatz (Eq. 9) with constants gamma4=1.01, gamma5=1.5, gamma6=13.8 that are described as close to optimal on the same benchmark datasets, so those curves evaluate a tuned threshold. The noise estimator eta feeding that ansatz relies on an approximation (Appendix B.2) that is only argued for small symmetric noise and is visibly biased on DeepWeeds (intercept 0.207 in Fig. 9).\n\nWhat is actually new: the systematic 19-dataset evaluation of a simple silhouette-like quantity (the authors acknowledge the relation to Scheidegger et al.'s silhouette scores in Section 2.1), the finite-size scaling analysis, and the mislabel application with foundation-model features. The code is public. The robustness checks across CLIP and DINOv2 backbones are a real plus. The authors are also candid about limits: they state Eq. (9) is not a fundamental element of the method, that the ansatz overshoots for confidence-based and human-annotator noise, and that the scaling analyses are not needed for the main difficulty estimate.\n\nThe soft spots line up with the stress-test. The headline correlation is moderate (rho=0.639, R^2 about 0.41) and only reaches 0.780 after post hoc exclusion of Oxford Flowers; the explanation is plausible but still post hoc. The finite-size decomposition claim is built on a rational-function fit with a fixed exponent (gamma0=1.808) over 8 datasets; high R^2 on fits does not by itself justify the \"first time this separation is possible\" phrasing. If the threshold constants or the eta approximation do not transfer to new datasets, the F1-level SOTA claim collapses, leaving the AUROC evidence, which is more modest but threshold-free.\n\nWho this is for: data-centric AI and dataset curation people, especially anyone building mislabel-detection pipelines. It deserves a serious referee. My recommendation: send it to review, but expect the authors to validate the threshold constants on held-out datasets or to demote the F1 claim in favor of the AUROC results. The core difficulty measure is worth engaging with.","headline":"Cheap, plausible dataset-difficulty measure with solid threshold-free AUROC evidence on hard datasets, but the headline F1 mislabel claim rests on thresholds tuned on the same benchmarks and needs held-out validation.","tokens_in":25604,"tokens_out":3620,"would_cite":true,"duration_ms":33289,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Class-wise reconstruction-error ratios on frozen features measure classification difficulty and catch label mistakes without training a classifier.","keywords":["reconstruction error ratios","classification difficulty","mislabel detection","label noise","class-wise autoencoders","foundation model features","dataset analysis","data-centric AI"],"falsifier":"Run the exact pipeline on a held-out benchmark suite not among the 19: if $\\chi$ does not rank datasets by their actual state-of-the-art error rates (Pearson correlation near zero), the difficulty claim fails; and if per-sample $\\chi$ gives chance-level AUROC for detecting corrupted labels at 20% symmetric noise, the mislabel claim fails.","tokens_in":24601,"feed_emoji":"📊","tokens_out":8199,"duration_ms":73107,"temperature":0.7,"pith_summary":"This paper tries to establish that a single family of cheap quantities—reconstruction error ratios (RERs) from tiny autoencoders trained per class on frozen foundation-model features—can characterize a classification dataset at three levels at once: sample, class, and whole dataset. Across 19 visual datasets, the average RER correlates with the error rate of state-of-the-art classifiers, and the paper shows the difficulty decomposes into a finite-sample-size part and a part due to Bayes error and decision-boundary complexity. Reinterpreting each sample's RER as a label-mistakenness score, the same pipeline detects corrupted labels at state-of-the-art F1 and AUROC levels for symmetric and asymmetric noise on hard datasets, using a threshold formula and a noise-rate estimator rather than training a classifier. If correct, the payoff is a data-curation tool that measures how hard a dataset is and which samples to reannotate, in seconds to minutes on CPU, from embeddings that already exist.","feed_headline":"Per-class autoencoder ratios predict difficulty and catch mislabels","feed_subtitle":"One cheap pass over frozen features measures dataset difficulty and flags mislabeled samples.","key_machinery":"The machinery is the class-wise reconstructor pair and the ratio built from it. Each class $c$ gets a shallow encoder-decoder trained on that class's frozen features with a UMAP graph-layout regularization term, so that reconstruction error $\\Delta_c(x)$ measures how well $x$ fits class $c$'s learned manifold. The sample-level score is $\\chi(x_c) = \\Delta_c(x_c)/\\min_{c'\\neq c}\\Delta_{c'}(x_c)$; values above 1 mean some other class's reconstructor represents the sample better than its own label does. The dataset-level determinant is the expectation $\\chi = \\mathbb{E}[\\chi]$, and two auxiliary ratios, $\\chi_0 = \\mathbb{E}[\\Delta_{\\tilde{c}}(x_{\\tilde{c}})/\\Delta_{\\text{rand}}]$ and $\\chi_{\\text{rand}} = \\mathbb{E}[\\Delta_{\\text{best}}/\\Delta_{\\text{rand}}]$, feed a noise-rate estimator $\\eta \\approx (\\chi_0 - \\chi_{\\text{rand}})/(1-\\chi_{\\text{rand}})$. For binary mislabel prediction, a hand-set threshold ansatz $\\chi^* = \\gamma_4 \\chi^{-\\gamma_5}/(1+\\gamma_6 \\eta_0)$, with $\\gamma_4 = 1.01$, $\\gamma_5 = 1.5$, $\\gamma_6 = 13.8$, converts the per-sample score into clean/dirty predictions. The same per-sample scores also convert into mistake probabilities via Bayes' rule with kernel density estimates.","core_discovery":"The paper's central object is the reconstruction error ratio $\\chi(x_c) = \\Delta_c(x_c)/\\min_{c'\\neq c}\\Delta_{c'}(x_c)$, where $\\Delta_c(x_c)$ is the reconstruction error of a 'reconstructor'—a shallow autoencoder with a UMAP graph-layout loss term—trained on the frozen CLIP or DINOv2 features of class $c$, evaluated on a sample labeled $c$. The denominator is the best reconstruction the sample gets from any other class's reconstructor. Averaged over a dataset, $\\chi = \\mathbb{E}[\\chi]$ serves as a dataset-difficulty determinant: across 19 visual datasets spanning four domains, $\\log(1-\\text{Accuracy})$ of state-of-the-art models correlates with $\\chi$ with Pearson $\\rho = 0.639$ ($\\rho = 0.780$ after removing Oxford 102 Flowers), and the same ratio, evaluated per sample, ranks how mistaken a label is. The paper argues that RERs decompose difficulty into a finite-sample-size term, extrapolated by fitting $\\chi_n = \\chi_\\infty (n^{\\gamma_0}+\\gamma_1)/(n^{\\gamma_0}+\\gamma_2)$, and a term capturing Bayes error and decision-boundary complexity, and that per-sample $\\chi$ with a threshold ansatz $\\chi^* = \\gamma_4 \\chi^{-\\gamma_5}/(1+\\gamma_6 \\eta_0)$ yields state-of-the-art mislabel detection under symmetric and asymmetric noise on hard datasets.","pith_inferences":["Because the per-sample RER is a distance-based ratio in feature space, it likely approximates a classification-margin signal; formalizing that connection could let RERs inherit margin theory and predict which samples will move under distribution shift.","The same pipeline should apply to text, audio, and time-series by swapping the foundation model, offering a uniform data-curation check across modalities rather than only vision.","The threshold-free AUROC evidence is the part of the mislabel claim that does not depend on the hand-set constants; a practical deployment could use RER rankings plus a small annotated validation set to pick a threshold, rather than the fixed ansatz.","Pairing RER mistakenness scores with a classifier-based method like Confident Learning could combine the class-decomposed manifold signal with inter-class boundary information, potentially covering the confidence-based noise regime where RERs are weaker."],"forward_implications":["Dataset difficulty becomes measurable without training a classifier: after one embedding pass, computing $\\chi$ takes seconds to minutes on CPU, so any team with frozen features can rank datasets by expected error rate.","Because $\\chi_n$ scales predictably with samples per class, the framework yields concrete estimates of how much accuracy is lost by pruning data or gained by collecting more annotations.","Per-sample RERs rank label mistakes, and with the threshold ansatz they match or beat Confident Learning and zero-shot baselines on symmetric and asymmetric noise on hard datasets, at realistic noise rates up to 30%.","The scores also convert into mistake probabilities, which allow budgeted reannotation decisions rather than only a clean/dirty binary flag.","The approach transfers across feature backbones (CLIP and DINOv2) and to out-of-domain medical datasets, suggesting the framework is not tied to a particular embedding model."],"supporting_citations":[{"why":"Supplies the CLIP ViT features that the reconstructors are trained on; these foundation-model embeddings make the difficulty probe comparable across all 19 datasets.","marker":"(Radford et al., 2021)"},{"why":"Supplies DINOv2 features used to show that RER difficulty estimates and sample rankings are robust to the choice of foundation model.","marker":"(Oquab et al., 2024)"},{"why":"Provides the UMAP graph-layout loss that regularizes the reconstructors so they learn both local and global structure of each class manifold.","marker":"(McInnes et al., 2018)"},{"why":"Provides the ParametricUMAP implementation used to train the autoencoders with UMAP regularization in the RER pipeline.","marker":"(Sainburg et al., 2021)"},{"why":"Confident Learning is the main feature-based mislabel detection baseline that RERs are compared against, and its reported label-error rates motivate the data-cleaning problem.","marker":"(Northcutt et al., 2021)"},{"why":"SimiFeat is the other feature-based mislabel detection baseline that RERs are compared against and outperform in the evaluated regimes.","marker":"(Zhu et al., 2021)"},{"why":"Provides the feature-based Confident Learning recipe and the four label-noise generation protocols (symmetric, asymmetric, confidence-based, human annotator) used in the mislabel evaluation.","marker":"(Srikanth et al., 2023a)"},{"why":"Supplies the three-source decomposition of classification difficulty (Bayes error, decision-boundary complexity, small sample size) that RERs claim to disentangle.","marker":"(Ho & Basu, 2002)"},{"why":"Probe nets are the prior dataset-difficulty estimation approach that RERs are compared against and improve on in speed and correlation with state-of-the-art error rates.","marker":"(Scheidegger et al., 2021)"}],"fun_headline_variants":["Autoencoder error ratios predict difficulty and catch mislabels","Reconstruction ratio probe flags mislabeled samples and rates datasets","Per-class autoencoder ratios measure classification difficulty","Ratio of class autoencoders: a cheap probe for label mistakes","Class-wise autoencoder ratios decode dataset difficulty and mistakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a sample is, on average, reconstructed better by its own class's autoencoder than by any other class's, and the headline mislabel-detection scores additionally assume a fixed threshold formula whose constants were tuned to the same benchmark datasets.","fun_headline_variants_meta":{"raw":{"variants":["Autoencoder error ratios predict difficulty and catch mislabels","Reconstruction ratio probe flags mislabeled samples and rates datasets","Per-class autoencoder ratios measure classification difficulty","Ratio of class autoencoders: a cheap probe for label mistakes","Class-wise autoencoder ratios decode dataset difficulty and mistakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3855,"prompt_tokens":1005,"completion_tokens":2850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":2772}},"tokens_in":621,"tokens_out":2850,"duration_ms":20481,"temperature":1.0,"reasoning_tokens":2772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:17:16.351001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact pipeline on a held-out benchmark suite not among the 19: if $\\chi$ does not rank datasets by their actual state-of-the-art error rates (Pearson correlation near zero), the difficulty claim fails; and if per-sample $\\chi$ gives chance-level AUROC for detecting corrupted labels at 20% symmetric noise, the mislabel claim fails.","supporting_citations":[{"cited_title":"Umap: Uniform manifold approximation and projection","cited_arxiv_id":null,"evidence_quote":"Provides the UMAP graph-layout loss that regularizes the reconstructors so they learn both local and global structure of each class manifold."},{"cited_title":"Parametric umap embeddings for representation and semisupervised learning","cited_arxiv_id":null,"evidence_quote":"Provides the ParametricUMAP implementation used to train the autoencoders with UMAP regularization in the RER pipeline."},{"cited_title":"Detecting corrupted labels without training a model to predict","cited_arxiv_id":null,"evidence_quote":"SimiFeat is the other feature-based mislabel detection baseline that RERs are compared against and outperform in the evaluated regimes."},{"cited_title":"Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy","cited_arxiv_id":null,"evidence_quote":"Probe nets are the prior dataset-difficulty estimation approach that RERs are compared against and improve on in speed and correlation with state-of-the-art error rates."}],"review_version":1}