{"id":"3adf0c33-153f-4693-a9a4-754dd99498a4","arxiv_id":"2608.02690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"GLOBE selects compact training subsets by matching multi-checkpoint gradient trajectories and their second-order statistics under structured sparsity, outperforming prior coreset methods on six image benchmarks.","lead":"GLOBE is a new method for choosing a small subset of training images that keeps model accuracy high. It tracks how each image's gradients change during training and picks a balanced, sparse set that mimics the full dataset's gradient behavior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GLOBE's reported gains may be confounded by its knowledge-distilled proxy: gradient baselines are not shown to use the same proxy or checkpoints, so the improvements may reflect proxy quality rather than the proposed trajectory matching.","rationale":"The reader identified teacher-proxy alignment as the weakest assumption. I partially agree, but I would sharpen the concern: the more dangerous version is not only that proxy gradients may not represent target gradients; it is that the experimental comparison does not control for proxy quality across methods. The method section gives no implementation details for baselines, so GLOBE's KD-aligned proxy is an uncontrolled variable. Because GLOBE's margins over baselines are small and its per-component ablations are also modest, a confounded proxy choice could fully explain the headline 'consistently outperforms.' This is an empirical concern, not a mathematical inconsistency. Other issues such as missing error bars, absent code, and the omitted budget constraint in Eq. 18 also matter, but the proxy confound is the one that can flip the central claim if it lands. A single controlled rerun of the strongest gradient baselines with GLOBE's proxy would settle it. I keep the reader's CONDITIONAL verdict: the method is plausible and the reported results are encouraging, but the claim needs this control before being accepted.","tokens_in":13547,"tokens_out":10540,"duration_ms":107843,"concrete_test":"Rerun Table 1's CIFAR-10 10% retention setting with CRAIG, GradMatch, and GLISTER using the exact GLOBE protocol: the same KD-initialized proxy network, the same T checkpoints, and the same class-balanced budgets. If these baselines rise toward GLOBE's 86.56 rather than staying near their reported 61.21, 60.10, and 56.86, the reported gains are a proxy artifact rather than evidence for trajectory-aligned matching.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that GLOBE's trajectory matching and structured sparsity beat existing gradient-based coreset selectors. The evaluation does not isolate that mechanism. CRAIG, GradMatch, and GLISTER require a proxy network, but Table 1 reports no protocol for the proxy architecture, teacher, distillation temperature, or checkpoint schedule used for baselines. GLOBE, by contrast, adds a KD-based teacher-proxy alignment step (Eq. 4) and multi-checkpoint trajectories. If baselines used vanilla proxies while GLOBE used a better distilled proxy, the comparison conflates method with proxy quality. The paper itself notes in 'Teacher-proxy alignment' that output-space KD does not explicitly align intermediate features or gradients, yet asserts it provides 'a reliable basis' without measuring proxy-target gradient similarity. The margins in Table 1 are modest (about 1.2 to 3.9 points), and the trajectory ablation alone accounts for only 1.68 points on the same setting, so a proxy confound is large enough to change the conclusion. Before accepting the contribution, we need a controlled comparison with the same proxy and trajectory data for all gradient-based baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GLOBE, a coreset selection method that represents each training sample by its gradient trajectory across multiple checkpoints of a proxy model, matches first-order (mean) and projected second-order statistics of the full-data trajectory distribution, and learns sparse nonnegative sample weights via Group LASSO and Elastic Net. The final coreset is obtained by class-balanced Top-K on the optimized weights. The authors evaluate GLOBE on six image classification benchmarks (CIFAR-10, CINIC-10, SVHN, ImageNet-100, ImageNet-1K, CIFAR-100) at 10%, 20%, and 30% retention ratios and report that it outperforms existing coreset methods in all 18 settings.","tokens_in":13870,"tokens_out":9637,"duration_ms":77328,"significance":"If the empirical claims are fully supported, GLOBE would be a strong new baseline for gradient-based coreset selection, with a clean convex formulation (modulo the budget-constraint issue discussed below), a practical multi-checkpoint trajectory representation, and a teacher-proxy distillation mechanism that addresses the proxy gap. The paper's strengths include the explicit optimization of a sparse weighting problem rather than greedy selection, and the use of second-order moment matching over trajectories. However, the current manuscript does not yet support the headline claim of consistent superiority because of evaluation gaps and a formulation discrepancy.","major_comments":[{"comment":"Equation (18) solves min_{ω≥0} of the matching loss plus regularizers, but it contains no simplex or budget constraint on ω, despite the abstract and Section 'Structured Sparse Optimization' stating that nonnegative budget constraints are used. Without a constraint such as Σ_i ω_i = K (or ≤ K), the solution's sparsity is determined only by the regularization constants, and the subsequent Top-K selection in Eq. (20) is not the argmin of the optimization problem that was solved. Please either add the budget constraint to the objective or revise the description of the method.","section":"Structured Sparse Optimization, Eq. (18)"},{"comment":"Table 1 reports accuracy averaged over the five evaluation architectures but gives no standard deviations, number of seeds, or per-architecture breakdown. The reported margins over the strongest baseline are modest (1.23–3.93 percentage points), so run-to-run variance could change the ranking. Add error bars and a table (or figure) with per-architecture results to support the claim that GLOBE consistently outperforms existing methods.","section":"Table 1"},{"comment":"The comparison against gradient-based baselines is confounded by the proxy protocol. GLOBE uses the knowledge-distillation objective in Eq. (4), whereas the manuscript does not specify whether CRAIG, GradMatch, and GLISTER used the same proxy architecture, teacher, distillation temperature, or checkpoint schedule. If baselines used undistiiled proxies, the observed gains may reflect the KD alignment rather than trajectory matching or structured sparsity. Please run all gradient-based methods on the same proxy trajectories and report the full protocol.","section":"Teacher-proxy alignment; Table 1"},{"comment":"The paper acknowledges that the output-space KD in Eq. (4) does not explicitly align intermediate features or gradients, yet it asserts that this provides 'a reliable basis' for constructing gradient trajectories. This assertion is load-bearing because the entire selection depends on proxy gradients. Provide a quantitative measure of proxy–target gradient similarity (e.g., cosine similarity between proxy and target gradients or trajectory agreement) across the five architectures to validate the assumption.","section":"Teacher-proxy alignment"},{"comment":"The ablation study does not isolate the second-order matching term (L_sec in Eq. (13)), which is a central component of the multi-order matching claim. Moreover, the final-checkpoint ablation (88.53 vs. 90.21) shows that the trajectory component contributes only 1.68 points in this setting, while removing Elastic Net or Group LASSO produces much larger drops (9 and 4.89 points, respectively). This suggests the regularizers, not the trajectory alignment, drive most of the improvement; ablating first-order vs second-order matching and reporting the result would help disentangle the contributions.","section":"Table 2"}],"minor_comments":[{"comment":"The second-order term is called 'covariance' in the Introduction, while Eq. (10)-(13) define it as an uncentered second moment; please make the terminology consistent throughout.","section":"Introduction and 'Multi-Order Distribution Matching'"},{"comment":"The section heading contains a typo; it should read 'Experiment and Analysis'.","section":"Section 'Experiment and Aanlysis'"},{"comment":"The horizontal axis label reads 'Smaples Within Group'; it should be 'Samples Within Group'.","section":"Figure 3(a)"},{"comment":"The reference list contains two entries, Cui et al. 2023a and 2023b, that appear to be the same paper (same title, authors, and venue); please merge or disambiguate them.","section":"References"},{"comment":"The proximal-gradient solver is described only briefly; please provide the initialization, convergence criterion, and the distribution of the random projection matrices R_j, as well as the values of γ_1, γ_2, T, m, M, τ, and η used in the experiments.","section":"Implementation details"},{"comment":"Table 1's formatting appears to omit entries for the 'Full' and 'GLOBE' columns at the 20% and 30% rows for several datasets; if this is a rendering artifact, please clarify the table layout.","section":"Table 1 formatting"}],"recommendation":"major_revision","confidential_remarks":"The paper is interesting and addresses a timely problem, but the evaluation is not yet convincing. I recommend a major revision. The missing budget constraint in the optimization and the lack of controlled comparison across proxies are the most serious issues. The authors should also verify that the reported table formatting and full-data accuracies are correct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about GLOBE. First, it is a real new method, not a restatement: it builds gradient trajectories from multiple checkpoints, matches their first-order mean and projected uncentered second-order moments, and wraps that in Group LASSO plus Elastic Net. That combination is not in the prior literature. Second, the empirical case for it is weaker than the abstract suggests, and the main problem is not the math but the comparison protocol.\n\nWhat the paper does well: the method section is mostly clear, the idea of \"trajectory-aligned\" selection is a natural extension of GradMatch, and the experiments cover six datasets and five architectures. The ablation study shows the trajectory component matters: replacing it with a single final checkpoint costs 1.68 points on CIFAR-10 at 10%. That is enough to take the core idea seriously.\n\nNow the soft spots. The most serious is the proxy confound. GLOBE adds a knowledge-distillation step to align a lightweight proxy with a stronger teacher, and then uses the proxy's gradients to select the coreset. The baselines (CRAIG, GradMatch, GLISTER) also need a proxy network, but the paper never says whether those baselines got the same distilled proxy, the same checkpoints, or even the same proxy architecture. If they ran on a vanilla proxy while GLOBE ran on a distilled one, the reported gains of 1 to 4 points could come from the proxy improving, not from the trajectory matching. The paper admits KD aligns output distributions, not gradients, and offers no measurement of proxy-to-target gradient similarity. That makes the teacher-proxy alignment a load-bearing assumption with no direct support.\n\nThere are smaller but real issues: Eq. (18), the claimed overall objective, has only ω≥0; the \"nonnegative budget constraints\" from the abstract are not in the optimization, and the class-balanced Top-K step comes after. So the optimization does not impose the budget. The paper also slips between calling the second-order term \"covariance\" in the intro and \"uncentered second moments\" in the method, which is a wording problem, not a substantive one. And there are no error bars or per-architecture numbers in Table 1, so we cannot judge whether the 1-4 point margins are stable or noise.\n\nThe central idea may well be right. But as it stands, the evidence is conditional, and the proxy protocol needs to be fixed before the claims can be trusted. Send this to peer review; a serious referee can ask for a controlled comparison with identical proxies and checkpoints, error bars, and the code. I would not cite it yet, but I would keep it on my radar.","headline":"Plausible and novel combination of trajectory matching and structured sparsity, but the evaluation doesn't separate the new mechanism from a distilled-proxy confound, so the claims are not yet supported.","tokens_in":14358,"tokens_out":2205,"would_cite":false,"duration_ms":23574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coresets selected from multi-checkpoint gradient trajectories beat single-snapshot baselines across six image benchmarks and five architectures.","keywords":["coreset selection","gradient matching","gradient trajectories","structured sparsity","Group LASSO","Elastic Net","data-efficient learning","on-device training"],"falsifier":"A direct test would train a proxy and a target model, extract per-sample gradient trajectories from the same checkpoints for both, and measure the cosine similarity between proxy and target trajectories; if that similarity is low yet GLOBE still wins on downstream accuracy, then the proposed mechanism of faithful proxy-gradient alignment is not the true source of its gains.","tokens_in":1603,"feed_emoji":"🎯","tokens_out":2217,"duration_ms":54300,"temperature":0.7,"pith_summary":"The paper introduces GLOBE, a coreset selection method that chooses a small subset of real training images by matching the full dataset's gradient behavior across multiple training checkpoints rather than at a single snapshot. The central claim is that this trajectory-aware selection, combined with sparse weight optimization, consistently improves downstream test accuracy over existing coreset methods, with the largest gains at low retention ratios. A sympathetic reader would care because coreset selection is a practical way to train models under memory and compute constraints, and GLOBE argues that capturing the temporal evolution of gradients is key to making small subsets preserve the full training signal.","feed_headline":"Trajectory-matched coresets win all 18 evaluated settings","feed_subtitle":"Matching the full dataset's gradient path over training checkpoints beats single-snapshot selection at low data budgets.","key_machinery":"The central object is the gradient trajectory: for each sample and tracked layer, the per-sample gradients are collected from $T$ checkpoints during proxy training and concatenated into one high-dimensional vector. This trajectory encodes how the sample influences optimization across early, middle, and late training stages. The paper's two-level matching objective aligns the first-order mean of these trajectories and the projected uncentered second-order moments, using random projections to keep the second-order computation tractable. The weight optimization is carried by Group LASSO over class-wise $k$-means clusters of penultimate-layer features, plus Elastic Net for sample-level sparsity, solved by a proximal-gradient style algorithm.","core_discovery":"GLOBE represents each training sample by a gradient trajectory, i.e., the concatenation of its per-layer gradients computed at several checkpoints during proxy training. It then learns a nonnegative weight vector over all samples by jointly minimizing two reconstruction losses: a first-order loss $\\|A\\omega-b\\|_2^2$ that matches the weighted mean trajectory to the full-data mean, and a second-order loss $\\|B\\omega-c\\|_2^2$ that matches the projected uncentered second moments of trajectory distributions. Group LASSO and Elastic Net regularizers, along with a class-balanced Top-$K$ selection step, turn the continuous weights into a discrete coreset of size $K$. The paper reports that this method achieves the highest accuracy in all 18 evaluated settings (six datasets at 10%, 20%, 30% retention) across five evaluation architectures, with the strongest gains over prior methods at the 10% retention level.","pith_inferences":["The paper does not measure how closely proxy gradients match target-model gradients; an unstated assumption is that the teacher-to-proxy knowledge distillation makes the proxy's trajectory a faithful stand-in, and a testable extension would directly compare proxy and target gradient trajectories at matched checkpoints to verify this.","The second-order matching relies on independent random projections per layer; an implicit premise is that the projected uncentered moments preserve enough covariance structure, and one could vary the projection dimension $m$ to see whether downstream accuracy improves monotonically with it.","The accuracy gains are largest at the lowest retention ratio, which suggests the dynamic gradient signal matters most when the data budget is extremely tight, and this could motivate adaptive checkpoint schedules optimized specifically for 1%–5% retention budgets."],"forward_implications":["At 10% retention, GLOBE improves over the strongest baseline by 1.23 points on CIFAR-10, 1.87 points on CINIC-10, and 2.08 points on ImageNet-100, showing that low-budget coreset training can get closer to full-data accuracy.","Ablations attribute +1.68 points to using multi-checkpoint trajectories instead of only the final checkpoint, +4.89 points to Group LASSO, and +9 points to Elastic Net, indicating each component contributes materially to the final accuracy.","The method shows that distribution matching plus structured sparsity is a viable template for sample selection on datasets with strongly correlated samples, beyond the greedy or pursuit-based schemes used by prior gradient-matching coresets.","Class-balanced Top-K selection maintains category coverage, making the coreset usable for many-class datasets such as ImageNet-1K and CIFAR-100, where the per-class budget can be as small as a few dozen images."],"supporting_citations":[{"why":"Defines CRAIG, the single-snapshot gradient matching baseline that GLOBE directly extends and compares against.","marker":"(Mirzasoleiman, Bilmes, and Leskovec 2020b)"},{"why":"Presents GradMatch, the OMP-based gradient matching method whose weighted subset objective GLOBE generalizes to trajectories.","marker":"(Killamsetty et al. 2021a)"},{"why":"Introduces Elastic Net, the ℓ1+ℓ2 regularization that GLOBE adopts for sample-level sparsity and correlated-sample stability.","marker":"(Zou and Hastie 2005)"},{"why":"Introduces Group LASSO, the structured sparsity regularizer that GLOBE uses to prune whole sample groups.","marker":"(Yuan and Lin 2006)"},{"why":"Shows that matching full training trajectories is effective for dataset distillation and motivates GLOBE's multi-checkpoint gradient trajectories.","marker":"(Cazenavette et al. 2022)"},{"why":"Defines GLISTER, a gradient-based coreset baseline that GLOBE outperforms and whose bilevel optimization is compared against.","marker":"(Killamsetty et al. 2021b)"}],"fun_headline_variants":["GLOBE: trajectory-aligned gradient matching for coreset selection","Coreset selection that tracks training dynamics beats snapshots","GLOBE: wins all 18 settings with multi-order trajectory matching","Gradient trajectories, not snapshots, for superior coresets","Matching the full gradient path beats single-snapshot coresets"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The coreset is selected entirely from gradients of a lightweight proxy model, and the paper assumes these proxy gradients faithfully represent the optimization dynamics of the target architectures it evaluates; no experiment measures the gradient similarity between proxy and target.","fun_headline_variants_meta":{"raw":{"variants":["GLOBE: trajectory-aligned gradient matching for coreset selection","Coreset selection that tracks training dynamics beats snapshots","GLOBE: wins all 18 settings with multi-order trajectory matching","Gradient trajectories, not snapshots, for superior coresets","Matching the full gradient path beats single-snapshot coresets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1584,"prompt_tokens":999,"completion_tokens":585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":496}},"tokens_in":615,"tokens_out":585,"duration_ms":5843,"temperature":1.0,"reasoning_tokens":496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:08:46.099503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would train a proxy and a target model, extract per-sample gradient trajectories from the same checkpoints for both, and measure the cosine similarity between proxy and target trajectories; if that similarity is low yet GLOBE still wins on downstream accuracy, then the proposed mechanism of faithful proxy-gradient alignment is not the true source of its gains.","supporting_citations":[],"review_version":1}