{"id":"6e558138-29a0-4c9b-9b19-e3aec03ec41d","arxiv_id":"2507.02979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"IMET, a hybrid of equal-class and misclassification-weighted sampling, reports 80.3% and 90.2% accuracy on OCTMNIST and PneumoniaMNIST with a 30k-parameter CNN.","lead":"The paper introduces IMET, a training method that repeatedly mixes class-balanced samples with samples the model misclassified in the previous round. On two MedMNIST benchmarks it reports higher accuracy than ResNet baselines using a much smaller CNN, but the evidence lacks error bars and full training details.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No matched random-sampling baseline and no error bars; the OCTMNIST margin is ~27 images and within sampling noise, so IMET's advantage is not established.","rationale":"The reader's weakest assumption is essentially the same one I would make: the paper does not isolate the IMET selection mechanism. I add two specifics. First, the OCTMNIST accuracy gap is within sampling noise for n=1,000, so even the headline comparison to ResNet is not statistically established. Second, the sample-count accounting is internally inconsistent with Algorithm 3, so the 'fewer training samples' part of the claim needs clarification. Both issues are addressable with a controlled re-implementation, which is why I would keep the CONDITIONAL verdict rather than reject outright. If the random-budget control matches IMET, the paper's contribution reduces to class-balanced resampling; if it does not, the claim would be supported.","tokens_in":14673,"tokens_out":11087,"duration_ms":128883,"concrete_test":"Reimplement IMET exactly with stated Nrep and k, and run 10 seeds, recording test accuracy after every round. Then train the identical CNN once on a stratified random sample that matches IMET's final per-class counts and total size (87,000 for OCTMNIST, 2,800 for PneumoniaMNIST), using the same optimizer, epochs, and test split. If the random-sample control's mean accuracy is within a 95% CI of IMET's mean, or if the reported number was the best round rather than the last round, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that IMET 'outperforms' the ResNet baselines rests on Tables 2-3, which report single-run accuracies on 1,000- and 624-image test sets with no error bars or seeds. For OCTMNIST, 80.3% vs 77.6% is a 27-image difference; the approximate standard error of the difference is 1.8 percentage points, so the 95% CI includes zero. More decisively, no control trains the same 30,047-parameter CNN on a random subsample of the same size and class composition as IMET. The closest in-house baselines are Baseline CNN (full 97,477 samples, 69.3%) and Equal Class Sampling (31,016 samples, 76.0%), so the 80.3% result could come from class-balanced resampling, from more total samples, or from chance rather than from misclassification-guided selection. Algorithm 3 also never specifies Nrep, k, the seed rule, or how the final round is chosen; if 80.3% is the best round on the test set, that is test-set selection. Separately, Table 2's '87,000' is hard to reconcile with Algorithm 3: with m≈7,900, one IMET round contains at most about 63,000 samples, so the reported count likely is cumulative across rounds and is not directly comparable with ResNet's 97,477-sample training set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Iterative Misclassification Error Training (IMET), a training procedure that alternates equal-class sampling with misclassification-weighted sampling, and evaluates it on OCTMNIST and PneumoniaMNIST using a small 30,047-parameter CNN. The paper reports that IMET reaches 80.3% and 90.2% accuracy on the two test sets, compares these numbers with MedMNIST ResNet baselines, and concludes that IMET outperforms the benchmarks while using fewer training samples. The algorithmic idea is clearly described, but the experimental evidence as presented does not establish the central claim.","tokens_in":14941,"tokens_out":8060,"duration_ms":89222,"significance":"IMET is a plausible data-selection heuristic that could be useful for small, imbalanced medical image datasets if its reported advantage is real. The manuscript includes comparisons with two in-house sampling baselines and with published ResNet results, and it reports several metrics (accuracy, precision, recall, F1, AUC) on standardized MedMNIST benchmarks. However, the evaluation consists of single-run point estimates without uncertainty quantification, lacks a matched random-sampling control, and leaves key algorithm parameters unspecified. The strength of the claim that IMET 'outperforms' the ResNet benchmarks is therefore out of proportion to the evidence presented.","major_comments":[{"comment":"The claimed advantage over the ResNet baselines is within sampling noise. For OCTMNIST, 80.3% versus 77.6% on the 1,000-image test set is a difference of 27 correct images; the approximate standard error of the difference is about 1.8 percentage points, so the 95% confidence interval includes zero. For PneumoniaMNIST, 90.2% versus 88.6% on the 624-image test set is a 1.6-percentage-point difference with an approximate standard error of about 1.8 points. The paper reports no seeds, no repeated runs, no error bars, and no significance tests. Without these, the conclusion in Section V that IMET 'successfully outperform[ed]' the OCTMNIST and PneumoniaMNIST benchmark models is not supported.","section":"Section IV-D, Tables 2-3"},{"comment":"No control trains the same 30,047-parameter CNN on a random subsample matched to IMET's training-set size and class composition. The in-house baselines (Equal Class Sampling and Weighted Sampling) differ from IMET in both the selection rule and the sample count (31,016, 63,000, and 87,000 samples for OCTMNIST; 2,428, 4,700, and 2,800 for PneumoniaMNIST), so the higher IMET accuracy could be explained by sample size or class balancing rather than by misclassification-guided selection. A random-subsample baseline matched to IMET's final training distribution is needed to isolate the effect of the IMET selection mechanism.","section":"Tables 2-3, Section IV-A"},{"comment":"The algorithm leaves Nrep, k, and the final-model selection rule unspecified, so the experiment is not reproducible and the reported sample counts are ambiguous. For OCTMNIST, where the smallest class in the training split is roughly m≈7,900, one round of Algorithm 3 contains at most about 5m≈39,500 samples (4m from the equal-class part plus at most m from the misclassified-weighted part), so the reported 87,000 samples in Table 2 cannot be a single-round training set. If it is a cumulative count across rounds, then '87,000 samples' counts repeated presentations of the same images and is not comparable with the ResNet's 97,477 single-pass training samples used in the Section V comparison. The authors should report Nrep, k, the per-round training-set sizes, the stopping criterion, and how the final model was selected; if the final round was chosen by test-set accuracy, that selection would be test-set leakage.","section":"Algorithm 3, Section III-A"},{"comment":"The blanket statement that IMET outperforms the PneumoniaMNIST benchmark models is contradicted by the AUC column of Table 3: IMET has AUC 0.895, while all four ResNet baselines have AUC between 0.944 and 0.962. Even if the accuracy point estimate is higher, the conclusion that IMET 'successfully outperform[ed]' the benchmarks is not justified when a threshold-independent metric is substantially worse. The authors should either temper the claim or explain why accuracy should be privileged over AUC in this comparison.","section":"Table 3, Section V"}],"minor_comments":[{"comment":"The abstract contains 'senarious' (should be 'scenarios'), and Section IV-C uses 'Receiving Operator Curve' (should be 'Receiver Operating Characteristic').","section":"Abstract, Section IV-C"},{"comment":"The input line of Algorithm 2 and Algorithm 3 uses 'inital' (should be 'initial'), and Algorithm 3 line 10 has 'fromdk' with a missing space, making the instruction harder to parse.","section":"Algorithm 2, Algorithm 3"},{"comment":"The caption states 'From right to left: CNV, DME, Drusen, and normal retina'; please confirm that the direction matches the order of the images in the figure.","section":"Figure 4 caption"},{"comment":"The 'Samples' column should explicitly state whether the number refers to unique training samples, cumulative presentations across rounds, or the final round's training-set size; the current ambiguity directly affects the data-efficiency claim.","section":"Tables 2-3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early-stage algorithmic proposal with a clear central idea, but the empirical validation is currently too thin for the strength of the claims. The missing controls and unspecified hyperparameters are fixable within the paper's scope, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: IMET is a sensible hybrid of two established ideas, but the central claim that it outperforms ResNet baselines is not supported by the experiments as reported. The margins are within sampling noise, and there is no control that trains the same model on a random subsample of the same size and class composition.\n\nWhat is new and good: Algorithm 3 is clearly specified, the paper correctly credits the two underlying techniques (equal class sampling and misclassification-weighted sampling), and it evaluates on standard MedMNIST benchmarks. The writing is accessible, and the conclusion honestly notes that the method has only been tested on medical images. The 50/50 mix is a legitimate variation on the cited methods, even if not a large one.\n\nThe soft spots are real and load-bearing. Every reported accuracy is a single run with no error bars, seeds, or significance tests. On OCTMNIST's 1,000-image test set, 80.3% versus 77.6% is a 27-image difference; the approximate standard error of the difference is 1.8 percentage points, so the 95% confidence interval includes zero. On PneumoniaMNIST's 624-image test set, 90.2% versus 88.4% is an 11-image difference. The closest in-house baselines are not matched: Baseline CNN uses all 97,477 samples, Equal Class Sampling uses 31,016, and Weighted Sampling uses 63,000, but IMET is reported at 87,000. Without a random-sampling control at the same sample count and class balance, the improvement cannot be attributed to the misclassification-guided selection. The sample counts are also internally confusing: with the smallest OCT class at 8,866, one IMET round contains at most about 71,000 samples, so the reported 87,000 is likely cumulative across rounds and is not directly comparable to ResNet's 97,477 unique training images. Finally, Nrep, k, and the seed rule are unspecified, and if the reported accuracy is the best round chosen on the test set, that is a form of test-set selection.\n\nThis paper is for a reader who wants a clearly written, preliminary demonstration of a cheap training heuristic. The idea is plausible and cheap to test properly, but the current evidence supports \"promising heuristic\" rather than \"outperforms ResNet.\" I would send it to peer review because the issues are fixable and a good referee could extract a solid contribution: seeds, error bars, a matched random-sampling baseline, an ablation of the 50/50 ratio, and a correct accounting of training samples. As it stands, the verdict should be major revision or reject, not accept.","headline":"IMET is a plausible hybrid of equal-class and misclassification-weighted sampling, but the reported accuracy gains over ResNet baselines are within sampling noise and lack the matched random-sampling baseline needed to support them.","tokens_in":15488,"tokens_out":3599,"would_cite":false,"duration_ms":40641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 30,047-parameter CNN retrained on its own misclassifications reports higher accuracy than ResNet-50 on two medical image benchmarks, using fewer training samples.","keywords":["Iterative Misclassification Error Training","IMET","curriculum learning","coreset selection","medical image classification","OCTMNIST","PneumoniaMNIST","class imbalance"],"falsifier":"Run IMET and a random-sampling control with the same 9-layer CNN, optimizer, epochs, retraining rounds, and per-round training budget on OCTMNIST and PneumoniaMNIST. If the random control reaches or exceeds IMET's reported 80.3% and 90.2% accuracies, the claimed benefit of misclassification-weighted selection is not real.","tokens_in":14451,"feed_emoji":"🩻","tokens_out":7394,"duration_ms":78348,"temperature":0.7,"pith_summary":"The paper proposes Iterative Misclassification Error Training (IMET), a way of deciding which training images to show a CNN on each round. After an initial training pass, IMET evaluates the model on a balanced subsample of the smallest class, counts the misclassified images per class, and builds the next training set from two halves: equal numbers of samples per class, plus samples drawn from misclassified pools in proportion to each class's error rate. The paper's claim is that this hybrid selection rule lets a small 30,047-parameter CNN outperform the ResNet-50 baseline on two MedMNIST benchmarks while using fewer training images, with reported accuracies of 80.3% versus 77.6% on OCTMNIST and 90.2% versus 88.6% on PneumoniaMNIST. The reason this would matter is that medical datasets are often small, imbalanced, and noisy, so a training rule that gets more accuracy from less data could make diagnostic models more practical.","feed_headline":"Small CNN beats ResNet-50 on two medical image sets","feed_subtitle":"IMET retrains on misclassified samples to reach 80.3% and 90.2% accuracy using far fewer images.","key_machinery":"The load-bearing object is the IMET data-update loop (Algorithm 3). It uses a balanced sub-sample $d_{sub}$ of training data, sized by the smallest class, to measure per-class misclassification counts; converts those counts into percentage weights $p_k$; and for each class draws $p_k\\%$ of samples from the misclassified pool and $m$ equal samples from the class, so the next epoch's training set is a 50/50 mixture of error-focused and class-balanced data. The 9-layer CNN with 30,047 parameters is the testbed, but the mechanism that carries the claimed gain is this iterative resampling rule, not the architecture.","core_discovery":"On the paper's own terms, the central discovery is that repeatedly retraining on the mistakes the model still makes, balanced against a fixed floor of equal class sampling, produces a more accurate classifier than standard full-data training with much larger networks. IMET keeps a balanced evaluation subset whose size is set by the smallest class, computes per-class misclassification rates on that subset, then forms each new training set as a 50/50 blend: half equal samples per class and half samples drawn from the misclassified images with probability proportional to class error. This is intended to keep attention on rare and edge-case classes without the oscillation that pure error-proportional weighting can cause. The paper reports that this procedure outperforms both equal-class and weighted sampling on OCTMNIST and PneumoniaMNIST, and matches or exceeds the accuracy and AUC of the ResNet-18 and ResNet-50 baselines with roughly 366-765 times fewer parameters and with thousands fewer training samples.","pith_inferences":["A direct test that would sharpen the claim is to compare IMET against random sampling with the same per-round sample size and training budget; without that control, the tables do not by themselves show that the selection rule, rather than the sample count, drives the accuracy gain.","The 50/50 blend invites a sweep of the mixture ratio; the optimal balance may lie elsewhere, and the paper gives no evidence that 50 percent is the right operating point.","Because IMET's evaluation subset is capped by the smallest class, scaling to large, highly imbalanced datasets may require an alternative error estimate; a validation-set-based or per-class accuracy estimate could be tested as a substitute."],"forward_implications":["A training rule that needs fewer images could lower the data and compute barrier for medical imaging models, where annotated data is scarce and expensive.","Small models trained with IMET could replace large pretrained backbones on MedMNIST-style tasks, making deployment on resource-constrained hardware more feasible.","Since IMET weights classes by their current misclassification rate, it should shift training toward rare and edge-case classes on imbalanced datasets.","The authors state the technique is not domain-specific and could be tested on transformers and large language models as a data-efficiency strategy."],"supporting_citations":[{"why":"It supplies the OCT and PneumoniaMNIST image datasets used for evaluation.","marker":"[15]"},{"why":"It provides the MedMNIST v2 benchmark and the ResNet baseline results that IMET is compared against.","marker":"[35]"},{"why":"It defines curriculum learning as a reweighting of the training distribution, the conceptual starting point for IMET.","marker":"[3]"},{"why":"It surveys curriculum learning and motivates training models from easier to harder samples.","marker":"[31]"},{"why":"It is a second survey of curriculum learning used to position IMET's difficulty-based training.","marker":"[27]"},{"why":"It describes misclassification-based weighted sampling for imbalanced data, which IMET extends by blending with equal sampling.","marker":"[24]"},{"why":"It formulates coreset selection as the search for a small representative subset, the subset-selection idea IMET adapts.","marker":"[21]"},{"why":"It supports the claim that curriculum-style training can improve deep networks, motivating the comparison with static coresets.","marker":"[10]"}],"fun_headline_variants":["IMET: Retraining on mistakes helps small CNN beat ResNet","Small CNN with error-focused training surpasses ResNet-50","IMET's mistake-driven retraining beats bigger CNNs on medical images","Error retraining makes small CNN match ResNet accuracy with fewer data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim assumes the accuracy gain comes from IMET's selection rule, but the experiments do not fix how many retraining rounds are run, how much training happens per round, or how the final model is chosen, and they do not compare against randomly chosen training samples of the same size.","fun_headline_variants_meta":{"raw":{"variants":["IMET: Retraining on mistakes helps small CNN beat ResNet","Small CNN with error-focused training surpasses ResNet-50","IMET's mistake-driven retraining beats bigger CNNs on medical images","Error retraining makes small CNN match ResNet accuracy with fewer data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":2138,"prompt_tokens":1007,"completion_tokens":1131,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":1065}},"tokens_in":623,"tokens_out":1131,"duration_ms":9660,"temperature":1.0,"reasoning_tokens":1065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:17:28.352904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run IMET and a random-sampling control with the same 9-layer CNN, optimizer, epochs, retraining rounds, and per-round training budget on OCTMNIST and PneumoniaMNIST. If the random control reaches or exceeds IMET's reported 80.3% and 90.2% accuracies, the claimed benefit of misclassification-weighted selection is not real.","supporting_citations":[{"cited_title":"S., Goldbaum, M., Cai, W., V alentim, C","cited_arxiv_id":null,"evidence_quote":"It supplies the OCT and PneumoniaMNIST image datasets used for evaluation."},{"cited_title":"Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification","cited_arxiv_id":null,"evidence_quote":"It provides the MedMNIST v2 benchmark and the ResNet baseline results that IMET is compared against."},{"cited_title":"Curriculum learning","cited_arxiv_id":null,"evidence_quote":"It defines curriculum learning as a reweighting of the training distribution, the conceptual starting point for IMET."},{"cited_title":"A survey on curriculum learning","cited_arxiv_id":null,"evidence_quote":"It surveys curriculum learning and motivates training models from easier to harder samples."},{"cited_title":"T., Rota, P ., and Sebe, N","cited_arxiv_id":null,"evidence_quote":"It is a second survey of curriculum learning used to position IMET's difficulty-based training."},{"cited_title":"Machine learning from imbalanced data sets 101","cited_arxiv_id":null,"evidence_quote":"It describes misclassification-based weighted sampling for imbalanced data, which IMET extends by blending with equal sampling."},{"cited_title":"On the power of curriculum learning in training deep networks","cited_arxiv_id":null,"evidence_quote":"It supports the claim that curriculum-style training can improve deep networks, motivating the comparison with static coresets."}],"review_version":1}