{"id":"b62cfdf3-9e4e-4710-9d5d-0a1928c64d58","arxiv_id":"2502.06905","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A lightweight score combining prediction mean and variance, plus ratio-adaptive Beta sampling, prunes datasets early in training and reaches 60% ImageNet accuracy at 90% pruning.","lead":"Researchers introduce DUAL, a dataset pruning score that combines example difficulty with prediction uncertainty, computed after only 30 to 60 training epochs. On CIFAR and ImageNet-1k they report state-of-the-art accuracy at high pruning ratios while cutting score-computation cost to 15%-66% of full-training methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DUAL score's marginal contribution at the headline 90% ImageNet operating point is unmeasured; Beta sampling alone may account for most or all of the gain, since the ImageNet random+β control is not reported.","rationale":"The paper's strongest claim is the ImageNet 90%-pruning result, and the condition that must hold for that claim to support the DUAL score is that the score, not just the Beta sampler, contributes materially to the gain. The authors' own CIFAR ablations (Table 4) show that Beta sampling alone accounts for a large share of the improvement and that a poor score like EL2N becomes competitive when combined with Beta sampling. This makes the missing ImageNet random+β control the most load-bearing unaddressed issue. The reader's weakest_assumption focuses on whether early-training prediction mean/variance orders examples by end-of-training usefulness; that is a related mechanism-level concern, but the attribution gap is more directly tied to the headline number and is testable with a single control experiment. The verdict remains CONDITIONAL: the central result is plausible and well-supported on CIFAR, but the ImageNet attribution needs this control before the DUAL score can be credited with the SOTA result. No ad hominem or theatrical language is intended; the concern is about experimental attribution, not author conduct.","tokens_in":40667,"tokens_out":15307,"duration_ms":140019,"concrete_test":"Run one ImageNet-1k 90%-pruning experiment with ResNet-34 using the paper's Beta sampling but with selection probability proportional only to the Beta PDF of prediction mean (no DUAL multiplier), training the selected subset for the same 300,000 iterations as the reported experiments. Compare top-1 accuracy to the reported 60.0%; include EL2N+β as a second score-agnostic control. If random+β reaches roughly 58% or EL2N+β matches 60%, the DUAL score is not the load-bearing component of the headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that DUAL plus Beta sampling reaches 60.0% top-1 accuracy on ImageNet-1k at 90% pruning, outperforming CCS (57.3%) and D2 (55.6%). The paper never isolates the contribution of the DUAL score at this operating point. Its own Table 4 shows that on CIFAR-10/100 at 90% pruning, replacing the DUAL score with random scores in Beta sampling yields 83.31/51.76, while DUAL+β yields 87.09/54.54; the Beta sampling alone adds 6.67 points over random on CIFAR-100, and DUAL adds only 2.78 points over random+β. Table 4 also shows EL2N+β reaches 87.00 on CIFAR-10, nearly identical to DUAL+β, indicating the score is almost irrelevant at extreme pruning. Because no random+β or EL2N+β control is run on ImageNet, the 7.7-point gain over random pruning in Table 2 cannot be attributed to the DUAL score rather than to the Beta sampling component. If random+β alone matches or exceeds CCS on ImageNet, the paper's central contribution reduces to Beta sampling, which is already close to BOSS (Acharya et al., 2024). This is a load-bearing gap in the argument for DUAL as the key novel component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DUAL (Difficulty and Uncertainty-Aware Lightweight), a per-sample score for dataset pruning defined as the product of (1 - mean prediction probability) and the standard deviation of prediction probabilities over a sliding window, averaged over an early training phase (30 epochs on CIFAR, 60 epochs on ImageNet-1k). To mitigate the accuracy collapse at high pruning ratios, the paper adds a Beta-distribution-based sampling scheme whose parameters depend on the pruning ratio so as to progressively include easier samples. Experiments on CIFAR-10/100, ImageNet-1k, label-noisy, image-corrupted, long-tailed, and cross-architecture settings report that DUAL plus Beta sampling achieves state-of-the-art accuracy at lower score-computation cost than full-training baselines, with the headline result being 60.0% top-1 accuracy on ImageNet-1k at 90% pruning using ResNet-34 after 60 epochs of score computation. A toy theoretical analysis (Theorem 3.1) claims that the DUAL score's ordering of two examples flips earlier in training than the Dyn-Unc variance score's ordering.","tokens_in":41031,"tokens_out":17544,"duration_ms":145455,"significance":"If the headline results hold, this is a genuinely useful contribution: it makes dataset pruning cheaper than full training and attains state-of-the-art accuracy at extreme pruning ratios on ImageNet-1k. The empirical package is broad and mostly careful: CIFAR experiments are multi-seed with error bars; the hyperparameters T, J, and c_D are ablated (Figure 6, Figure 9); and the evaluation spans noise, corruption, long-tailed distributions, cross-architecture transfer, and a comparison with dynamic pruning methods (Appendix B.5). The release of code is a further strength. The main weakness is attribution: the paper's own ablations (Table 4) show that the Beta sampler, not the DUAL score, carries most of the benefit at 90% pruning, and the corresponding control on ImageNet is missing. In addition, the closest baseline for the sampling component (BOSS) is never benchmarked, and the theoretical proof in Appendix D contains an informal small-learning-rate step. These gaps need to be closed before the central claim is fully established.","major_comments":[{"comment":"The headline ImageNet result does not isolate the contribution of the DUAL score. Table 4 shows that at 90% pruning the Beta sampler, not the score, is the dominant factor: on CIFAR-100, random thresholding gives 45.09 while random+β gives 51.76 (+6.67), and the DUAL score adds only 2.78 points over random+β (54.54 vs 51.76); on CIFAR-10, EL2N+β (87.00) is within the reported error bars of DUAL+β (87.09). Table 2 reports no random+β control on ImageNet, so the 7.7-point margin of DUAL+β over Random at 90% pruning (60.0 vs 52.3) cannot be attributed to the DUAL score; since random+β gains 6.67 points over Random on CIFAR-100, random+β alone could plausibly reproduce most of the ImageNet gain. Furthermore, the DUAL score used as a plain thresholding rule at 90% on ImageNet (53.1) is barely above Random (52.3) and well below CCS (57.3), so the state-of-the-art result is created by the sampler rather than by the score. The authors should report random+β, and ideally EL2N+β and Dyn-Unc+β, on ImageNet at 80% and 90% pruning before claiming that the DUAL score is what makes the method state-of-the-art.","section":"§4.2 (Table 2), §4.4 (Table 4)"},{"comment":"The strict inequality T_vm < T_v is not proved as stated. The decisive step is the final, unquantified condition: \"If the learning rate is sufficiently small, then γ_V(t) cannot significantly increase in one step, allowing γ_V(t) to fall between C^{-1} and 1.\" This condition is exactly what converts the weak consequence T_vm ≤ T_v into the strict T_vm < T_v, and no bound on η or on the per-step increment of γ_V is provided. The supporting material is also informal: Lemma D.7 is invoked as \"Theorem D.7\", and Equation (18) is introduced with \"assume for now\" and justified by \"this generally holds for sufficiently small η\". Since the main text labels Theorem 3.1 as \"Informal\", one option is to state the strict-flip claim as a conjecture supported by the empirical figures; if it is to be called a theorem, the small-learning-rate condition needs a quantitative statement.","section":"§D.1 (proof of Theorem 3.1)"},{"comment":"The closest existing method to the Beta-sampling half of the contribution, BOSS (Acharya et al., 2024), is discussed in a Remark but is never included in any comparison table. The Remark claims that the non-linear parameterization (r^{c_D}) and the use of the prediction mean rather than a difficulty score as the PDF argument give DUAL's sampler an advantage over BOSS's mode-linear scheme, but no experiment supports this claim. Given that the paper's own ablations indicate that the sampler carries most of the benefit at high pruning ratios, BOSS is a necessary baseline; without it, the novelty and superiority of the sampling component are unverified.","section":"§3.3 Remark; §4.1 Baselines; §4.2-4.4"}],"minor_comments":[{"comment":"Table 2 reports single-run ImageNet results without error bars; the headline margin over D2 (60.0 vs 55.6 at 90% pruning) should be accompanied by at least two or three runs or an explicit statement that these are single-run results, since the CIFAR tables report five-run averages.","section":"Table 2"},{"comment":"The reproduced Dyn-Unc baseline is far below Random at high pruning ratios (Table 1: CIFAR-10 at 90% = 59.67 vs Random 83.74), which is at odds with the strong results reported in the original Dyn-Unc paper; please verify that the all-window averaging in Eq. (2) and the hyperparameters match the original Dyn-Unc implementation, since a mismatch would inflate the apparent advantage of DUAL over Dyn-Unc in Tables 1 and 2.","section":"§4.2, Table 1; Eq. (2)"},{"comment":"Appendix A.1 states that TDDS hyperparameters for ImageNet 80% and 90% pruning were \"set arbitrarily\" because the original paper provides no settings; those rows in Table 2 should be marked as such, or the arbitrary choices documented in the table caption, to avoid giving the impression that these are the original method's reported values.","section":"Appendix A.1 (TDDS)"},{"comment":"Notation and typos: in Eq. (3), k ∈ [T−J+1] should read k ∈ {1, …, T−J+1}; Table 2's caption contains \"raio\"; Figure 5's caption contains \"mislabeld\"; and Appendix D refers to \"Theorem D.7\" where the statement is labeled \"Lemma D.7\".","section":"Eq. (3); Table 2 caption; Figure 5 caption; Appendix D"},{"comment":"The values of c_D are set per dataset and per noise level (Appendix A.2: c_D = 6 for 20%/30% noise, 8 for 40% noise, and 2 for Tiny-ImageNet corruption); please state how these were selected (for example, on a held-out validation set) so that the favorable noise-robustness results cannot be read as per-condition oracle tuning, even though Figure 6 shows robustness across c_D ∈ {3, …, 6} in the clean setting.","section":"Appendix A.2, §B.1-B.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the empirical package is solid, but I would not accept it without the ImageNet attribution control described in major comment 1, and I would ask the authors to double-check the Dyn-Unc reproduction before the revision is returned, since its underperformance relative to Random at high pruning ratios on both CIFAR and ImageNet is surprising given the original paper's claims. If the control runs show that random+β matches DUAL+β on ImageNet, the paper should be reframed around the Beta-sampling component and must then benchmark against BOSS."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical contribution is real: computing a pruning score after 30-60 epochs instead of full training, and using a prediction-mean Beta sampler to avoid the accuracy cliff at high pruning ratios. The DUAL score itself—(1 - mean prediction) × prediction std—is a small but sensible twist on Dyn-Unc, and the efficiency claims (15% of CIFAR cost, 66% of ImageNet cost) are attractive if they replicate. The paper is also unusually honest in its ablations: Table 4 shows how much of the gain comes from the sampling alone, and the long-tailed, noise, corruption, cross-architecture, and dynamic-pruning comparisons give the empirical core real breadth. The toy theorem in Appendix D is fine as intuition, and clearly labeled as such.\n\nThe soft spots are proportionate but real. The main one is exactly what the stress-test flags: at the headline 90% ImageNet operating point, there is no random+β or EL2N+β control. On CIFAR-100 at 90%, random+β gets 51.76 and DUAL+β gets 54.54, so DUAL adds only 2.78 points; EL2N+β on CIFAR-10 nearly matches DUAL+β. Without the control on ImageNet, the 60.0% result cannot be assigned to the DUAL score rather than to the Beta sampling mechanism. That gap matters because the paper frames DUAL as the novel component, and Beta sampling is already close to BOSS. Also, the reproduced Dyn-Unc baseline underperforms random on ImageNet, which is suspicious and suggests the baseline may have been run with poor settings. ImageNet numbers are single-run, with no error bars. The \"first metric to combine difficulty and uncertainty\" phrasing is overstated given SIMS and BOSS, though the specific combination is new.\n\nNone of this is fatal. The central claim—that lightweight early-training pruning can be state-of-the-art at extreme ratios—is broadly supported on CIFAR and likely true on ImageNet, but the attribution of the gain needs one more experiment. The paper deserves a serious referee; I would send it out with a request for the missing ImageNet control and multi-seed runs. My own verdict would be conditional rather than clear accept.","headline":"Useful, honest empirical paper whose headline ImageNet gain is not yet attributable to the DUAL score rather than the Beta sampler; worth refereeing.","tokens_in":41489,"tokens_out":1479,"would_cite":true,"duration_ms":16228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that dataset pruning can be made lightweight by computing a DUAL score—difficulty times uncertainty—in the first 60 epochs, plus a Beta-sampling selection step that keeps test accuracy at 60% on ImageNet-1k even after…","keywords":["dataset pruning","coreset selection","training dynamics","prediction uncertainty","example difficulty","Beta sampling","early stopping","image classification"],"falsifier":"Run the same DUAL+$\\beta$ procedure on ImageNet-1k with a different architecture family (e.g., ViT or a ConvNeXt) while keeping the score-computation epoch at 60: if the accuracy at 90% pruning drops below the random baseline (52.3% on ResNet-34), the early-to-late ranking transfer breaks and the central claim fails to generalize. A second falsifier would be to compute the Spearman rank correlation between DUAL scores at epoch 60 and Dyn-Unc scores at epoch 90 on ImageNet-1k; if that correlation is close to zero, the 'moon-shaped' argument loses its empirical footing.","tokens_in":40508,"feed_emoji":"✂️","tokens_out":2011,"duration_ms":22163,"temperature":0.7,"pith_summary":"This paper aims to show that dataset pruning does not need to be more expensive than training itself, a paradox that plagues many existing methods. The authors propose a new scoring metric—DUAL—that combines example difficulty and prediction uncertainty, computed during only the early phase of training, so that the pruning decision is made before a full training run finishes. They also add a pruning-ratio-aware sampling step using a Beta distribution to avoid collapsing test accuracy when most of the dataset is discarded. The result, if correct, would make selective dataset reduction a practical, cheap preprocessing step for large-scale learning, which matters because training-cost savings are exactly the point of pruning.","feed_headline":"Prune datasets in a third of the time and keep accuracy","feed_subtitle":"A difficulty-uncertainty score plus Beta sampling beats full-training baselines at 90% pruning on ImageNet.","key_machinery":"The core object is the DUAL score, defined as $$\\mathrm{DUAL}_k(x,y) := \\left(1-\\bar{P}_k\\right)\\, \\sqrt{\\frac{\\sum_{j=0}^{J-1}\\left(P_{k+j}(y\\mid x)-\\bar{P}_k\\right)^2}{J-1}}$$ where $\\bar{P}_k$ is the average prediction probability in a window of length $J$, so the score is difficulty ($1-\\bar{P}_k$) times uncertainty (the standard deviation). This single number is averaged over all windows up to the computation epoch $T$, and it is used to give each sample a sampling probability via a Beta distribution whose parameters are linear functions of the pruning ratio and of the dataset complexity hyperparameter $c_D$. The Beta sampling step is what turns a naive hard-sample selection into a hybrid difficulty-diversity method at high pruning ratios, and the toy-theorem in Appendix D shows that the DUAL score flips the ordering of two separable points strictly earlier than plain variance does.","core_discovery":"The central claim is that a single score, the Difficulty and Uncertainty-Aware Lightweight (DUAL) score, can rank examples by their eventual usefulness during the full training run while being computed only 30–60 epochs into a 90–200 epoch schedule. The score is the product of (1 − mean prediction probability) times the standard deviation of predictions within a sliding window; the paper argues that this product identifies uncertain, hard-but-learnable samples earlier than Dyn-Unc's plain variance, because the difficulty factor corrects for the fact that early variance is dominated by easy samples that are still being learned. The paper also claims that threshold pruning at aggressive ratios is fundamentally limited, and that a Beta-distribution sampling step, whose shape depends on the pruning ratio and on the estimated complexity of the dataset, selects a better-balanced subset at extreme ratios. The strongest evidence is the ImageNet-1k result at 90% pruning: 60.0% test accuracy with ResNet-34 using only 66% of the score-computation time of the previous state-of-the-art methods, and the observation that the method removes mislabeled data almost perfectly at low pruning ratios on noisy CIFAR-100.","pith_inferences":["The paper's moon-shaped trajectory argument suggests a testable prediction: for a fixed network family and dataset, the epoch at which DUAL scores stabilize (i.e., the ranking by DUAL at epoch $T_1$ and $T_2$ becomes highly correlated) should predict the best early-stopping point. No such 'stabilization curve' is reported, but the Spearman correlations in Appendix B already give the raw material.","The fact that Beta sampling rescues even random selection at 90% pruning suggests a more general principle: at extreme pruning ratios, the value of any score may be secondary to the sampling distribution, which would motivate evaluating DUAL's Beta shape against other parametric families (e.g., truncated Gaussian) as a direct ablation; the paper does not do this.","The toy theorem is stated for two points with exponential loss and gradient descent; extending the 'earlier flip' argument to stochastic gradients, finite windows $J>1$, and the noisy-label setting the paper emphasizes would be a natural next step, and if it fails, the empirical claims would need to be attributed to network specifics rather than to the score's geometry.","If DUAL+$\\beta$ is robust to label noise, then the same score, computed on a small warmup run, could be used to flag subsets for targeted human review before full training; this is a downstream application the paper does not discuss."],"forward_implications":["If DUAL's ranking is reliable at epoch 30–60, then pruning can be performed as a cheap preprocessing step: the total cost stays below a single full training run on the original dataset, which is what the current greedy baselines cannot achieve.","The noise experiments suggest that early DUAL scores separate mislabeled samples from genuinely hard samples, because noise is under-learned at that stage; this could make pruning a practical way to clean noisy datasets without explicit noise-detection training.","The cross-architecture results imply that a small model's early scores can be reused to prune for a larger model, further reducing the cost of pruning for architectures that are expensive to train from scratch.","The Beta-sampling step improves even random selection at 90% pruning on CIFAR-100, which, if correct, indicates that the biggest accuracy loss at high pruning ratios comes from representation bias—not from which score is used—and that the sampling rule itself is a reusable component for other scores."],"supporting_citations":[{"why":"Provides the theoretical motivation that Pareto-optimal pruning can convert power-law error scaling into exponential scaling, which is the background for why dataset pruning is worth doing.","marker":"Sorscher et al. (2022)"},{"why":"Defines the Dyn-Unc score that DUAL builds on; the paper's central claim is that DUAL outperforms Dyn-Unc when scores are computed early, so Dyn-Unc is both the baseline and the conceptual ancestor.","marker":"He et al. (2024)"},{"why":"EL2N establishes the early-training approach to importance scores; the paper compares against it and uses the same rank-correlation stability analysis.","marker":"Paul et al. (2021)"},{"why":"CCS is the previous state-of-the-art hybrid method at high pruning ratios on ImageNet-1k; DUAL+$\\beta$ claims to beat it, so CCS is the main numerical baseline.","marker":"Zheng et al. (2022)"},{"why":"D2 is the other hybrid high-ratio baseline that DUAL+$\\beta$ claims to beat on ImageNet-1k; it is cited in the introduction as a method that is expensive due to graph construction.","marker":"Maharana et al. (2023)"},{"why":"BOSS also uses a Beta distribution for difficulty-based sampling; the paper distinguishes its nonlinearity and the use of prediction mean as the basis of the PDF, so BOSS is the direct antecedent for the sampling part.","marker":"Acharya et al. (2024)"},{"why":"Dataset Cartography establishes the mean versus standard deviation plot of prediction probabilities as a way to visualize training dynamics; the moon-shaped trajectory is the motivation for the DUAL score.","marker":"Swayamdipta et al. (2020)"},{"why":"Provides the implicit-bias result for gradient descent on separable data that the toy theorem in Theorem 3.1 relies on to prove that the DUAL score flips earlier than variance.","marker":"Soudry et al. (2018)"}],"fun_headline_variants":["DUAL score prunes 90% of data, keeps accuracy, cuts time","Prune without full training: DUAL score keeps accuracy","Early-stage DUAL score cuts pruning time to 66% on ImageNet","Beta-sampled DUAL pruning: 60% accuracy at 90% pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ranking of samples by DUAL score computed in the first 30–60 epochs reliably matches the ranking that a full-training Dyn-Unc score would produce, so that samples flagged as uncertain early are the ones that would prove most informative at the end of training.","fun_headline_variants_meta":{"raw":{"variants":["DUAL score prunes 90% of data, keeps accuracy, cuts time","Prune without full training: DUAL score keeps accuracy","Early-stage DUAL score cuts pruning time to 66% on ImageNet","Beta-sampled DUAL pruning: 60% accuracy at 90% pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001405,"raw_usage":{"total_tokens":5708,"prompt_tokens":1006,"completion_tokens":4702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":4620}},"tokens_in":622,"tokens_out":4702,"duration_ms":30704,"temperature":1.0,"reasoning_tokens":4620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:48:27.249768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same DUAL+$\\beta$ procedure on ImageNet-1k with a different architecture family (e.g., ViT or a ConvNeXt) while keeping the score-computation epoch at 60: if the accuracy at 90% pruning drops below the random baseline (52.3% on ResNet-34), the early-to-late ranking transfer breaks and the central claim fails to generalize. A second falsifier would be to compute the Spearman rank correlation between DUAL scores at epoch 60 and Dyn-Unc scores at epoch 90 on ImageNet-1k; if that correlation is close to zero, the 'moon-shaped' argument loses its empirical footing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the theoretical motivation that Pareto-optimal pruning can convert power-law error scaling into exponential scaling, which is the background for why dataset pruning is worth doing."},{"cited_title":"Deep learning on a data diet: Finding important examples early in training","cited_arxiv_id":null,"evidence_quote":"EL2N establishes the early-training approach to importance scores; the paper compares against it and uses the same rank-correlation stability analysis."},{"cited_title":"Balancing feature similarity and label variability for optimal size-aware one-shot subset selection","cited_arxiv_id":null,"evidence_quote":"BOSS also uses a Beta distribution for difficulty-based sampling; the paper distinguishes its nonlinearity and the use of prediction mean as the basis of the PDF, so BOSS is the direct antecedent for the sampling part."},{"cited_title":"Dataset cartography: Mapping and diagnosing datasets with training dynamics","cited_arxiv_id":null,"evidence_quote":"Dataset Cartography establishes the mean versus standard deviation plot of prediction probabilities as a way to visualize training dynamics; the moon-shaped trajectory is the motivation for the DUAL score."}],"review_version":1}