{"id":"fb3b7a9c-0897-41d2-9d1d-f6de215b9510","arxiv_id":"2507.16729","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Tuning coreset sampling parameters on validation data improves downstream F1 and balanced accuracy, and can beat full-data training in several settings.","lead":"This paper tunes how training-data coresets are sampled, including per-class allocation and active sampling, to improve classification metrics rather than just loss. In experiments, tuned coresets often beat vanilla coresets and sometimes match or beat training on the full dataset, suggesting data selection matters more than data volume.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains come from the validation-argmax over a hundreds-configuration grid; without controlling for selection bias and a comparably tuned full-data baseline, the 'tuned beats full' claim is not yet supported.","rationale":"The paper's contribution is empirical: a systematic grid-search framework over coreset sampling parameters. To believe the central claim that tuned coresets significantly outperform vanilla coresets and full-data training, the evaluation must show that the selection process does not manufacture the gains. I agree with the reader that Proposition 1 is a genuine gap, but I do not think it is the single most load-bearing issue: even if Proposition 1 were proven, the logistic-regression, SVM, and active-sampling experiments would still be evaluated under the same selection protocol. Conversely, if the selection-bias check fails, the DTC proposition is moot. The paper deserves credit for its clear ablation visualizations and the concrete active-sampling algorithm, and the underlying idea is worth studying; however, the empirical central claim requires the distributional analysis described. I therefore keep the reader's conditional verdict without changing it, while identifying a different primary concern.","tokens_in":20659,"tokens_out":6547,"duration_ms":72325,"concrete_test":"Re-run Table 1 (A9A, unified/leverage/other logistic-regression coresets): for every configuration in the Table 6 grid, record validation F1 and test F1. Then (i) compute the Spearman correlation between validation and test F1 across configurations; (ii) compare the test F1 of the validation-argmax with the median test F1 over the grid and with the test F1 of a full-data logistic regression whose hyperparameters are themselves tuned over a comparable-size grid; (iii) run a permutation test that shuffles validation F1 labels to estimate how much test F1 gain a random selector would appear to achieve. If the validation-argmax test F1 is within one standard error of the grid median, or if the tuned full-data baseline closes the gap, the headline claim that tuned coresets outperform full-data training is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Operationally, 'tuned' means the single configuration that maximizes validation F1 over roughly 5 coreset sizes × 7 class allocations × 6 deterministic ratios × 3 weight strategies (Appendix A, Tables 6–7). Picking the argmax of many noisy validation estimates and then reporting only its test score is a textbook selection-bias setup: the maximum is expected to beat the typical configuration and often the true best configuration even when the tuning signal is pure noise. The paper reports no error bars, no repeated sampling seeds, and no comparison of the selected configuration to the distribution of test scores over the grid. It also compares this heavily searched coreset pipeline to an untuned full-data baseline: logistic and SVM models use a fixed regularization parameter and XGBoost uses library defaults, while the coreset side gets hundreds of configuration attempts. The active-sampling experiments (Table 5), where both sides receive hyperparameter tuning, show the advantage shrinking or reversing on several metrics (e.g., IEEE F1 74.34 vs 74.56; ROC AUC lower), which is consistent with the selection-bias reading. The DTC Proposition 1 concern raised by the reader is real, but the evaluation-protocol concern is more load-bearing: it threatens the abstract's broad claim across all classifiers, not just the XGBoost results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a practical framework for tuning coreset construction for classification tasks. It introduces three mechanisms on top of sensitivity-based coresets: deterministic inclusion of high-probability points with reweighting, class-wise allocation of the sample budget, and an active-sampling refinement loop (Algorithm 1). The authors report experiments on eight binary datasets using logistic regression, SVM, and XGBoost, and claim that tuned coresets significantly outperform vanilla coresets and often outperform training on the full dataset with respect to balanced accuracy, F1, and ROC AUC.","tokens_in":21010,"tokens_out":5488,"duration_ms":55225,"significance":"If the central claim holds, the paper would make a useful contribution to data-centric machine learning by showing that coreset generation can be optimized for downstream classification metrics rather than for loss approximation alone. The breadth of datasets and classifiers, and the ablation study that isolates each tunable parameter, are strengths. However, the main claim is not yet established by the evidence as presented: the evaluation protocol selects the best validation configuration over a large grid, the full-data baselines are not given an equivalent hyperparameter search, no error bars or repeated-seed results are reported, and the decision-tree experiments rely on an unproven proposition. With the current evidence, the paper is best read as a preliminary empirical report rather than a definitive demonstration.","major_comments":[{"comment":"The 'tuned' configurations are selected by taking the argmax of validation F1 over a grid of roughly 5 coreset sizes × 7 class allocations × 6 deterministic ratios × 3 weight schemes (Tables 6–7), but the paper reports no error bars, no repeated sampling seeds, and no distribution of test metrics over the grid. Moreover, the full-data baselines in Sections 4.1–4.3 use a fixed regularization parameter (Section 4.1: 'a regularization parameter equal to 1') and XGBoost library defaults, so the 'tuned beats full' comparison conflates data tuning with model selection. Because the maximum of many noisy estimates is biased upward even when the tuning signal is pure noise, the headline claim in the abstract is not supported unless the authors report the full grid distribution, repeat each configuration over seeds, and compare against a full-data baseline that receives an equivalent hyperparameter search. The results in Table 5, where both sides are tuned, show the advantage shrinking or reversing on several metrics (e.g., IEEE F1 74.34 vs 74.56; IEEE ROC AUC 95.81 vs 96.59), which is consistent with this concern.","section":"§4.1–4.3, Tables 1–3, Appendix A"},{"comment":"Proposition 1 states that a coreset for logistic regression is sufficient for decision tree classification 'from a practical point of view,' but no proof is given, and the cited fact that gradient boosting implements logistic regression additively does not imply that an ε-coreset for the logistic loss preserves the XGBoost training objective or the resulting classification metrics. Since the entire DTC section depends on this transfer, either replace the proposition with a precise formal statement and proof, or reclassify it as an empirical conjecture and weaken the claims accordingly.","section":"§4.3, Proposition 1"},{"comment":"The validation ROC AUC values of 3.77%, 3.75%, 3.64%, and 2.72% are not valid ROC AUC values for a binary classifier on a balanced dataset; an ROC AUC below 50% (and especially near 3%) is implausible unless the quantity being reported is something else. The test ROC AUC values in the same table are approximately 70%, so the table is internally inconsistent. This must be corrected, and the reported metric must be defined precisely, because Table 4 is one of the main pieces of evidence for the DTC claim.","section":"Table 4, Hepmass rows"},{"comment":"Algorithm 1's output is described as 'an ε-coreset ... that practically outperforms' the input coreset, but no argument shows that adding actively sampled points with weight 1 preserves the ε-coreset property; the 'better or same' property is only a consequence of the final validation comparison in Lines 18–22, not of coreset theory. The active sampling query is also unspecified: the manuscript does not name the active learning algorithm, the query size m, the patience ρ, or the final coreset sizes used in Table 5. Please specify these details and either prove or remove the ε-coreset guarantee from the output description.","section":"§3, Algorithm 1"}],"minor_comments":[{"comment":"The notation paragraph is incomplete: 'let Y denote the set of functions mapping instances from X to h' ends abruptly and never defines h or Y; please fix.","section":"§2.1"},{"comment":"Several figure captions list '(b)' twice (e.g., 'where (a) refers to using a coreset ratio of 0.5%, (b) 5.4%, (b) 10.3%, and (d) 20%'); the second '(b)' should be '(c)'.","section":"Appendix C, Figures 5–10"},{"comment":"The dataset names are used inconsistently: 'A9A' and 'cod-rna' appear alongside the names 'Adult' and 'CodRNA' used elsewhere; please unify the nomenclature.","section":"§4.1, Table 1"},{"comment":"No details are given for the active-sampling implementation in Table 5, including the specific query algorithm, the number of points m added per iteration, the patience ρ, the number of iterations until convergence, and the hyperparameter tuning protocol for the Full∗ and Active sampling∗ rows; these details are needed for reproducibility.","section":"§4.4, Table 5"},{"comment":"The DataHeroes framework [Dat22] appears to be a core component of the implementation, but the reference is only a company name without a version, documentation URL, or repository; please provide sufficient information for reproducibility, and state any commercial interest.","section":"References, [Dat22]"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The authors are affiliated with DataHeroes, Inc., and the paper evaluates a DataHeroes-based implementation; a clear statement of the commercial relationship and code availability would help reviewers assess reproducibility. The Hepmass ROC AUC anomaly and the selection-bias issue should be resolved before the paper can be considered for publication; if the authors cannot provide error bars and fair baselines, the central claim should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the interesting part is the demonstration that coreset construction parameters—deterministic sampling, class allocation, and weight handling—can shift classification F1 by several points, and the ablations show the effect is real within their setup. That is a legitimate empirical contribution for the coreset audience. The headline claim that tuned coresets beat full-data training, however, is not supported by the evidence as presented.\n\nWhat is actually new: each ingredient is known (FSS20 deterministic sampling, MTBR23 class allocation, active sampling), but the systematic combination and the study of their effect on downstream metrics rather than loss approximation is new. The experiments cover eight datasets and multiple coreset families; the Appendix C ablations are genuinely useful, showing that tuning one parameter often moves the metric and that the best parameter depends on the coreset type.\n\nThe soft spots are real and the stress-test note has it right. The tuned configuration is the argmax over a large validation grid (up to 5 coreset sizes, 7 class allocations, 6 deterministic ratios, 3 weight strategies); picking the max of many noisy estimates and reporting only that test score is textbook selection bias. The full-data baselines are not given the same search; logistic and SVM use a fixed regularizer, XGBoost uses library defaults. So the 'tuned beats full' claim conflates coreset tuning with the benefit of a grid search. The active-sampling table, where both sides receive hyperparameter tuning, shows the advantage shrinking or reversing on several metrics, which is consistent with that reading. Also: Table 4 reports ROC AUC values of 3.77% for Hepmass, which are impossible; that's a data-entry error but it undermines confidence. Proposition 1 is not a proposition; it is a heuristic justified by an appeal to the additive nature of gradient boosting, and the paper explicitly says it 'validates' it via experiments. The algorithm's output is called an ε-coreset without proof, since it adds points based on validation improvement.\n\nWho this is for: people building coreset pipelines for classification and anyone interested in data-centric ML. The idea is worth one revision. My recommendation: send to review, but the referee should demand error bars or repeated seeds, a comparably tuned full-data baseline, a corrected Table 4, and an honest downgrade of Proposition 1 to a conjecture. If the evaluation is fixed, the core message—that you can tune coreset sampling for classification metrics—will be solid. As it stands, the paper is a strong empirical cookbook that overstates its main claim.","headline":"An honest empirical survey of tuning coreset sampling for classification metrics, but the no-error-bars grid-search selection and untuned full baselines blow a hole in the 'tuned beats full' claim.","tokens_in":21439,"tokens_out":3968,"would_cite":false,"duration_ms":41229,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that tuning how a coreset is sampled—adding deterministic inclusion, class-wise budgets, and active refinement—produces small training sets whose models beat both vanilla coresets and, in most cases, full-dataset training.","keywords":["coresets","data-centric AI","training set selection","importance sampling","classification metrics","F1 score","active sampling","class imbalance"],"falsifier":"Train XGBoost on the same datasets with (a) the unified logistic-regression coreset of [TMF20], (b) an equal-size coreset built from a loss native to trees, such as the MSE-based decision-tree coreset of [JSSNF21], and (c) uniform sampling; if (a) does not match or beat (b) and (c) on validation and test $F1$, then Proposition 1 is false and the Section 4.4 active-sampling gains reduce to generic subset selection rather than coreset guarantees.","tokens_in":20494,"feed_emoji":"🎯","tokens_out":9759,"duration_ms":88730,"temperature":0.7,"pith_summary":"Coresets are weighted subsets of a training set chosen to reproduce the full-data training loss, and the paper's starting point is that reproducing the loss is not the same as training a good classifier. The authors argue that the coreset recipe itself can be tuned for the classifier you actually want, and they introduce three adjustable mechanisms: deterministically including the highest-probability points to restore diversity, allocating the sample budget across classes by hand instead of by class frequency, and refining the subset with active sampling that queries the points the current model is most unsure about. Across logistic regression, SVM, and XGBoost on eight datasets, the tuned coresets outperform vanilla coresets on metrics such as $F1$ and balanced accuracy, and in most experiments they also outperform models trained on the full data. If the results hold, data curation becomes a tunable part of model design: a smaller training set can be better than the full one, not just cheaper.","feed_headline":"Tuned coresets beat full-dataset training on classification metrics","feed_subtitle":"Sensitivity sampling plus class-aware and deterministic selection turns smaller training sets into better classifiers.","key_machinery":"The central object is the *tuned coreset*, built on a sensitivity-based $\\varepsilon$-coreset (a weighted subset drawn by importance sampling with probabilities derived from sensitivity scores) and modified by three mechanisms. Deterministic sampling takes the highest-probability points into the coreset exactly once and reweights the rest, following the diversity-preserving construction and Theorem 31 of [FSS20]. Class-wise allocation splits the sample budget across classes by a tunable ratio rather than the data's natural class frequencies. Active sampling (Algorithm 1) repeatedly trains the model on the current coreset, asks an active learner for the $m$ points the model is most uncertain about from the remaining training data, and adds them until a patience counter sees no validation-metric improvement; the algorithm returns whichever of the original or refined coreset scores better. Proposition 1—that a logistic-regression coreset is practically sufficient for decision-tree classification, justified by gradient boosting implementing logistic regression additively—is the bridge that lets the unified coreset of [TMF20] drive the XGBoost experiments.","core_discovery":"On the paper's own terms, the discovery is that sensitivity-based coreset construction has hidden tunable knobs—deterministic sampling, class-wise sample allocation, and deterministic weight-handling strategies (keep, inv, prop)—and that turning them with a validation-set grid search materially changes downstream classification quality. Tuned coresets are reported to beat their vanilla versions by roughly 2–3 percentage points or more in $F1$ score across logistic regression and SVM on the A9A and CodRNA benchmarks, and to match or exceed full-data training in most configurations. For XGBoost, active sampling extends the same gains to strongly imbalanced fraud-detection datasets (IEEE, Credit Card, E-commerce), where the refined coreset beats the full-data model on balanced accuracy and $F1$. The paper frames this as evidence that a coreset can be an opportunity to improve generalization and mitigate class bias, not merely an approximation device.","pith_inferences":["Implicit in the method is that coreset construction becomes a hyperparameter-optimization problem; the authors flag meta-learning or reinforcement learning for picking the sampling parameters as future work, which would make the tuning loop itself automatic.","If the central claim generalizes, the training value of a point is not uniform: minority-class points, points near the decision boundary, and high-sensitivity outliers carry more value per sample than the bulk, so any curation method that ignores class balance and diversity forfeits classification performance.","A testable extension the paper does not run is multiclass or regression analogues; if the same three knobs do not produce similar relative gains there, the effect may be specific to binary classification with $F1$-style metrics.","Because uniform random sampling is sometimes competitive with vanilla sensitivity coresets in the reported tables, a separate ablation that applies the same three knobs to a random (non-sensitivity) coreset would separate how much of the gain comes from the knobs and how much from the sensitivity theory."],"forward_implications":["Coreset size can be spent deliberately: shifting part of the sample budget toward minority classes improves balanced accuracy and $F1$ without changing the total subset size, and the paper notes this does not harm the $\\varepsilon$-coreset approximation guarantee.","Where vanilla sensitivity sampling is most degenerate—when a few points dominate the sampling probabilities—deterministic inclusion restores diversity, and the weight-handling strategies (keep, inv, prop) give a tunable trade-off between the deterministic and probabilistic parts of the coreset.","Active sampling converts a coreset from a static compression into an iterative refinement loop that, by construction, returns the better of the original or refined coreset with respect to the chosen metric on the validation set.","For tree ensembles, the results imply that a coreset designed for logistic loss can serve as a practical training subset for XGBoost, beating uniform sampling and, on imbalanced fraud data, beating full-data training on balanced accuracy and $F1$."],"supporting_citations":[{"why":"Defines sensitivity-based coresets and the approximation guarantee in (2) that all baselines inherit.","marker":"[BFL+16]"},{"why":"Supplies the deterministic-sampling mechanism and its weight adjustment, with Theorem 31 preserving approximation guarantees.","marker":"[FSS20]"},{"why":"The unified (f-SVD) coreset used for logistic regression, SVM, and as the logistic-regression coreset that Proposition 1 extends to XGBoost.","marker":"[TMF20]"},{"why":"Leverage-score logistic-regression coreset, one of the vanilla baselines the tuning must beat.","marker":"[MSSW18]"},{"why":"Lewis-weight logistic-regression coreset, another vanilla baseline.","marker":"[MOP22]"},{"why":"Monotonic-kernel coreset baseline for logistic regression.","marker":"[TJF22]"},{"why":"SVM coreset baseline, tuned in the SVM experiments.","marker":"[TBFR21]"},{"why":"XGBoost implementation that the decision-tree experiments train, whose additive-tree structure motivates Proposition 1.","marker":"[CG16a]"},{"why":"Shows gradient boosting implements logistic regression additively, the cited basis for Proposition 1.","marker":"[Fri01]"}],"fun_headline_variants":["Tuned coresets outclass full-data training in F1","Class-aware coreset tuning lifts model accuracy","Deterministic sampling sharpens coreset classifiers","Tuning coresets turns less data into better classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The XGBoost results depend on Proposition 1: that a coreset built for logistic regression is practically sufficient for decision-tree classification, which the paper supports by analogy to gradient boosting rather than by proof; if that bridge fails, the tree-boosting results lose their coreset grounding.","fun_headline_variants_meta":{"raw":{"variants":["Tuned coresets outclass full-data training in F1","Class-aware coreset tuning lifts model accuracy","Deterministic sampling sharpens coreset classifiers","Tuning coresets turns less data into better classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3393,"prompt_tokens":865,"completion_tokens":2528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2466}},"tokens_in":481,"tokens_out":2528,"duration_ms":20339,"temperature":1.0,"reasoning_tokens":2466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:03:20.608831+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train XGBoost on the same datasets with (a) the unified logistic-regression coreset of [TMF20], (b) an equal-size coreset built from a loss native to trees, such as the MSE-based decision-tree coreset of [JSSNF21], and (c) uniform sampling; if (a) does not match or beat (b) and (c) on validation and test $F1$, then Proposition 1 is false and the Section 4.4 active-sampling gains reduce to generic subset selection rather than coreset guarantees.","supporting_citations":[],"review_version":1}