Data Pruning by Information Maximization

Haoru Tan; Shizhen Zhao; Sitong Wu; Wei Huang; Xiaojuan Qi

REVIEW 4 major objections 4 minor 24 references

Data Pruning by Information Maximization

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that coreset selection is one discrete quadratic program: maximize sample importance minus redundancy. With graph-cut conditional gains it maximizes information, and beats prior methods at high pruning rates.

desk verdict InfoMax's empirical recipe is worth taking seriously, but its information-theoretic equivalence proof does not hold as written; the paper needs a fix or a reframe. read the letter →

arxiv 2506.01701 v2 pith:XAVTGLFP submitted 2025-06-02 cs.CV cs.AI

Haoru Tan , Sitong Wu , Wei Huang , Shizhen Zhao , Xiaojuan Qi This is my paper

classification cs.CVcs.AI

keywords datapruningcoresetselectiondiscretequadraticprogramminginformationmaximizationgraph-cutconditionalgainredundancyminimizationvision-languagepretraininginstructiontuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InfoMax sets out to show that data pruning, picking a small subset of a training set that preserves as much learning value as possible, is a single discrete optimization problem rather than a heuristic cut on scores or a clustering step. The objective is a quadratic program: maximize the sum of each selected sample's importance score minus a penalty for pairwise similarity within the selected set. The paper argues that with a graph-cut instantiation of conditional information gain, this program is exactly the set-level information-maximization problem behind coreset selection. It then gives an iterative softmax solver that makes the quadratic program tractable at million-sample scale. The payoff is that one formulation beats score-based, diversity-based, and greedy hybrid baselines on image classification, CLIP vision-language pretraining, and LLM instruction tuning, with the largest gains at aggressive pruning rates.

What carries the argument

The load-bearing object is the discrete quadratic program with a pairwise redundancy term, and the named instantiation that ties it to information theory is the Graph-cut conditional gain (GCCG), defined as $I(z_k \mid z_1,\dots,z_{k-1}) = I(z_k) - 2\lambda \sum_{i<k} K_{z_i,z_k}$, so a sample's marginal information after earlier picks is its own score minus a penalty proportional to summed similarity with earlier picks. Substituting GCCG into the chain-rule expansion of set information collapses $I(S)$ into a sum of individual scores minus pairwise similarities, which is exactly the quadratic objective; with $\alpha = \lambda/(p-1)$, solving the quadratic program is claimed to be equivalent to solving the information-maximization coreset problem. The solver is the proximal-gradient softmax update $X^{t+1} = \operatorname{Softmax}(p I - 2p\alpha K X^t)$, and $k$-nearest-neighbor sparsification plus random dataset partitioning reduce the quadratic cost and let the method run on millions of samples.

What would settle it

Take a trained classifier and the InfoMax coreset at a 10% selection ratio, then for several random orderings of the selected samples measure the actual conditional gain of adding each next sample (for instance by retraining on the growing subset and recording validation-loss reduction) and compare those numbers with the GCCG formula. If the formula orders the candidate additions differently from the measured gains, then the objective in Eq. (2) is not the set information in Eq. (1); a second check is to construct a dataset containing near-duplicate high-score samples and see whether the InfoMax objective correctly predicts that keeping both duplicates is worse than keeping one plus a mid-score sample.

Watch

Extended reading notes

Core claim

The central claim is that the optimal coreset of size $p$ is the maximizer of $\sum_{z\in S} I(z) - 2\lambda \sum_{z\neq s \in S} K_{z,s}$, and that this expression is not just a heuristic balance between importance and diversity but follows from the chain rule for set information when the conditional gain of adding a sample $z_k$ after $z_1,\dots,z_{k-1}$ is taken to be the graph-cut conditional gain $I(z_k) - 2\lambda \sum_{i<k} K_{z_i,z_k}$. Introducing binary selection variables $X_z$ turns this into the discrete quadratic program $\max_{X\in\{0,1\}^N,\, |X|=p}\; \sum_z X_z I(z) - \alpha \sum_{z,s} K_{z,s} X_z X_s$. The paper further claims that relaxing $X$ to $[0,1]^N$ and iterating the softmax update $X^{t+1}=\operatorname{Softmax}(p I - 2p\alpha K X^t)$ yields a practical selection rule, and that the coresets it selects train models that match or beat the previous best methods, especially when 90% to 99% of the data is pruned.

Load-bearing premise

The equivalence rests on treating the graph-cut conditional gain $I(z_k) - 2\lambda \sum_{i<k} K_{z_i,z_k}$ as a genuine conditional information gain inside the chain-rule expansion; the paper selects this instantiation rather than deriving it, so if that model of redundancy is wrong for real data, the claimed identity between the quadratic program and information-maximizing selection does not follow.

Editorial extensions

If this is right

At 10% selection on CIFAR-10, CIFAR-100, and ImageNet-1K, InfoMax improves accuracy over the best prior hybrid baseline by roughly 1.4 to 3.4 percentage points, and the gap widens as pruning becomes more aggressive.
With only unsupervised scores and features (SSP scores on DINO embeddings), InfoMax can approach or beat supervised baselines at high pruning rates, so training an extra surrogate model is not strictly required.
The same quadratic formulation transfers across modalities: it improves CLIP pretraining on CC12M at all tested selection ratios and improves LLaMA2-7B instruction tuning by up to about 5.5 percentage points on BBH at a 5% selection rate.
The solver is practical: roughly 16 minutes to select from one million images and about 37 minutes for 12 million image-text pairs.
The method is robust to its hyperparameters, with $k=5$, $\alpha=0.3$, $T=20$, and partition sizes above roughly one million samples giving consistently good results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper does not pursue: because the objective separates into a per-sample score and a pairwise similarity term, replacing $K$ with a stronger redundancy measure, such as a learned similarity or a kernel matched to the downstream task, should improve coreset quality without changing the solver.
The softmax update gives each sample a selection probability, so one natural extension is to sample the coreset from that distribution (for example with Gumbel-top-p) instead of taking the top-$p$ entries, which would expose selection uncertainty; the paper does not propose this.
If the GCCG equivalence is correct, the quadratic objective connects coreset selection to penalized-diversity optimization, so approximation guarantees for such programs could become relevant to data pruning; the paper itself does not develop this connection.
The recommendation that partitions stay above roughly one million samples implies that partitioning changes the composition of the selected coreset, so measuring how much the selected set varies across random partitions would quantify the method's stability; this is left implicit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the InfoMax recipe—importance plus pairwise redundancy as one quadratic objective, with a softmax iterative solver—is a real step forward for data pruning, and the empirical results are consistent enough to take seriously. But the paper's information-theoretic justification is currently unproven, and if you rely on it you will find the proof doesn't close.

What's genuinely new: prior hybrids like D2-Pruning use greedy graph message passing and CCS uses score binning; InfoMax instead solves a single DQP with an explicit redundancy term, and the sparsification + dataset partitioning make it scale to 12M samples in about 37 minutes. The experiments cover classification, CLIP pretraining, and instruction tuning, with gains that are largest at high pruning rates—exactly where data pruning matters. The ablations on k, d, α, T are honest and inform the chosen defaults. Code is released. That is a solid empirical contribution.

Where the paper wobbles: the central claim in Sec 3.3 that the DQP is equivalent to maximizing set-level information under graph-cut conditional gain does not hold as written. Substitute Eq (15) into Eq (14): you get I(S) = Σ f(z) − λ Σ_{z≠s} K_{z,s} for symmetric K. Eq (5) instead has λ/(p−1) on the ordered-pair sum; the factor of 2 from ordered vs unordered pairs is absorbed, but the (p−1) is not explained. The proof in Appendix D.2 does not repair this—its first line replaces the objective with a sum over all subsets contained in the selected set, which is a different quantity, and the factorial reductions that follow do not follow from that starting point. The solver also deserves a caveat: the softmax output sums to 1, not p, so the relaxed cardinality constraint is not preserved; taking the top-p is a reasonable heuristic, but the paper should say so. Baselines are reported without error bars while InfoMax's own five-run std is given, which makes the margin claims slightly less sharp.

None of this breaks the empirical recipe. The DQP objective is a reasonable a priori model, the downstream evaluations are genuine, and the gains are plausible. But the theory framing needs either a correct proof or a honest reframing as a heuristic objective.

Who should read this: anyone working on data-efficient training, coreset selection, or dataset curation. It is likely to be a useful baseline for future work. I would cite it. For peer review, yes—it deserves a serious referee, with the request to fix the equivalence claim before publication.

Referee Report

4 major / 4 minor

Summary. The paper proposes InfoMax, a coreset-selection method that formulates data pruning as a discrete quadratic program (DQP): it maximizes the sum of per-sample importance scores minus a pairwise redundancy penalty, subject to a cardinality constraint. A scalable gradient-based solver is derived via continuous relaxation, proximal-gradient steps, sparsification of the similarity matrix, and dataset partitioning. The authors claim that, under a graph-cut conditional gain (GCCG) instantiation, solving this DQP is equivalent to maximizing the set-level information defined in Eq. (1), and they support the method with experiments on image classification (CIFAR, ImageNet), vision-language pretraining (CC12M/CLIP), and instruction tuning (LLaMA2-7B), reporting consistent gains over score-based, diversity-based, and hybrid baselines, especially at high pruning ratios.

Significance. If the equivalence in Sec. 3.3 were established, InfoMax would provide a clean unifying objective that interpolates between importance-based and diversity-based pruning, and the paper's empirical contribution is substantial: the method is clearly specified, the code is released, the ablations cover the main hyperparameters, and the evaluation spans three task families with consistent wins at aggressive pruning rates, plus cross-model/cross-setting generalization and time-cost studies. The main theoretical claim, however, is currently not proven and, as written, contains coefficient inconsistencies; the significance of the paper therefore rests on the empirical method standing alone unless the information-theoretic derivation is repaired or explicitly repositioned as motivation rather than proof.

major comments (4)

[Sec. 3.3 / App. D.2 Eq. (16)] The central equivalence claim is not established because the coefficient in the set-level expression is wrong. Substituting the GCCG definition Eq. (15) into the chain rule Eq. (14) gives I(S) = sum_{z in S} I(z) - 2 lambda sum_{i<j} K_{z_i,z_j}. Since K is symmetric, sum_{i<j} K = (1/2) sum_{z != s in S} K_{z,s}, so the correct set-level expression is I(S) = sum_{z in S} I(z) - lambda sum_{z != s in S} K_{z,s}. Eq. (16) instead has coefficient 2 lambda on the unordered distinct-pair sum, and it places the sum over D without the indicator variables X, so as written I(S) in Eq. (16) does not depend on S. The subsequent conversion to Eq. (5) with coefficient lambda/(p-1) is therefore not a consequence of the chain-rule expansion. Concretely, for p=3 with I(z)=1 and all pairwise K=0.5, Eqs. (14)-(15) with lambda=1 give I(S)=0, while Eq. (5) with alpha=lambda/(p-1)=0.5 gives 1.5.
[Appendix D.2, Eqs. (17)-(19)] The proof in Appendix D.2 does not repair the gap. Its first line replaces max_{|S|=p} I(S) with max_X sum_{S subset D} (prod_{z in S} X_z) I(S), but the latter sums I(S) over every subset S contained in the selected set, not just the selected set itself; for a binary X with exactly p ones, the sum contains 2^p terms and is not equal to I(S(X)). The summation reductions in Eqs. (17)-(19), including the introduction of the factorial constants and the final use of (p-2)!/(p-1)! = 1/(p-1), are not valid algebraic consequences of the preceding line. Thus the advertised equivalence between Eq. (1) and Eq. (2) remains unproved.
[Sec. 3.3, Eq. (5)] Even if the GCCG instantiation is accepted, Eq. (5) does not match the DQP in Eq. (2) at the level of objective coefficients. Eq. (2) penalizes alpha sum_{z,s in D} K_{z,s} X_z X_s over ordered pairs (including the diagonal unless explicitly excluded), while Eq. (5) penalizes (lambda/(p-1)) sum_{z != s} K_{z,s} X_z X_s. For a symmetric kernel, the correct chain-rule reduction requires alpha = lambda/2 if the DQP sum is over ordered distinct pairs, or alpha = lambda if it is over unordered distinct pairs; neither equals lambda/(p-1) for general p. The statement 'if we set alpha = lambda/(p-1), then we obtain the quadratic programming problem' is therefore not a valid identification, and the p-dependence introduced by 1/(p-1) has no counterpart in Eq. (15).
[Sec. 3.3, unnumbered display after Eq. (4)] The text says the chain-rule equation 'always holds regardless of the order of samples,' but the conditional gain terms I(z_k | z_1, ..., z_{k-1}) are never defined as formal conditional information; they are simply the increments I({z_1,...,z_k}) - I({z_1,...,z_{k-1}}). With the fixed GCCG formula (15), this order-invariance should be verified, and the sense in which Eq. (4) is an information-theoretic chain rule should be stated explicitly.

minor comments (4)

[Algorithm 1, line 11] Line 11 says 'Append the samples corresponding to the top-k largest item in X^T', but within each partition the number of samples to select should be the per-subset budget p/d, not k; k is already used for the sparsification neighborhood size, so this is confusing and likely a typo.
[Eq. (2)] The sum over z,s in D in Eq. (2) includes the diagonal terms K_{z,z} X_z^2; the paper should state whether K is assumed to have zero diagonal or whether self-similarity is excluded, because the diagonal changes the gradient and the effective redundancy coefficient.
[Table 1] The CCS row for CIFAR-10 appears to have a formatting error ('95.495.093.0 91.0 86.9'), with a missing 100% entry and missing separators; please correct the table.
[Sec. 4.4] The ablation in Fig. 7(c) reports that alpha around 0.3 to 4 is satisfactory, but Eq. (5) suggests alpha should scale as 1/(p-1); a sentence explaining how a fixed alpha across different selection ratios is compatible with the proposed theoretical mapping would be helpful.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reported benchmark results are external downstream evaluations of an a priori objective, and the self-citations are not load-bearing. The information-theoretic equivalence in Sec. 3.3 is conditional on the GCCG ansatz and has proof defects, but it is a modeling tautology rather than a circular fit.

full rationale

The paper's empirical claims are self-contained against external benchmarks: InfoMax selects coresets with the objective in Eq. (2), then trains fresh models (ResNet, CLIP, LLaMA2-7B) on the selected subsets and measures held-out accuracy, retrieval scores, and F1. These evaluations are genuine downstream outcomes, not reconstructions of the selection inputs. Hyperparameters alpha, k, d, and T are tuned via ablations on CC12M and then reused; this is standard hyperparameter selection, not a fitted input renamed as a prediction. The self-citations (Tan et al. 2021 for the proximal-gradient solver; Tan et al. 2023 as one possible importance score) are not load-bearing: the solver derivation is also attributed to Baque et al., Krahenbuhl & Koltun, and Larsson et al., and the importance-score instantiation is interchangeable with SSP, EL2N, and other cited scores. The claimed equivalence in Sec. 3.3 and Appendix D.2 is explicitly conditional on choosing the Graph Cut conditional gain (GCCG) as an instantiation from the submodular information measures literature; the paper does not derive GCCG from first principles. Consequently, the equivalence between Eq. (1) and Eq. (2) is a consequence of the chosen information model, not an independent prediction. The proof also has algebraic gaps: Eq. (16) introduces a factor-of-two discrepancy relative to the chain-rule expansion, Eq. (5) introduces an unexplained 1/(p-1) factor, and Appendix D.2 asserts a factorial reduction without a complete derivation. These are correctness and rigor defects in a conditional justification; they do not make the central benchmark results circular. Overall, the paper's main empirical contributions are not equivalent to their own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its free parameters are standard hyperparameters selected by ablation. The main load-bearing assumptions are that GCCG represents conditional information gain and that inner-product similarity represents information overlap; both are plausible but not formally grounded.

free parameters (4)

α (pairwise redundancy weight) = 0.3
Chosen after ablation on CC12M zero-shot ImageNet (Fig 7c) and applied globally across CIFAR, ImageNet, and LLM tasks.
k (sparse neighborhood size) = 5
Set to 5 after ablation (Fig 7b) showing robustness; chosen for efficiency.
d (dataset partition factor) = 10 for CC12M
Chosen so each partition has at least 1M samples; ablation Fig 7a shows performance degrades below that scale. Not used on CIFAR/ImageNet.
T (solver iterations) = 20
Ablation Fig 7d shows gains saturate after 20 iterations; used in all experiments.

assumptions (4)

domain assumption Graph-cut conditional gain (GCCG) is a valid instantiation of the conditional information gain in the chain-rule expansion of set information.
Introduced in Sec 3.3, Eq (15), and used to derive Eq (16) and the claimed equivalence to Eq (2). No proof is given that GCCG has the required information-theoretic properties.
domain assumption Pairwise inner-product feature similarity K quantifies information overlap or redundancy between samples.
Used throughout Eq (2) and Eq (3); the paper states this as a premise in Sec 1, but does not justify it beyond intuition.
standard math The proximal gradient / entropic mirror descent solver converges and its softmax update solves the relaxed problem.
Invoked in Appendix D.1 via citations to Krähenbühl & Koltun 2011, Larsson et al. 2018, and Baqué et al. 2016; the O(1/T) convergence statement is cited, not proved.
domain assumption Pruning each random dataset partition independently preserves near-optimal coreset quality.
The dataset partition strategy in Sec 3.2 is validated empirically in Fig 7a, but no theoretical guarantee is provided that the union of per-partition coresets solves the global objective.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Data Pruning by Information Maximization." pith.science (2026). https://pith.science/paper/XAVTGLFP

@misc{pith2026250601701,
  author       = {Pith},
  title        = {Pith review of: Data Pruning by Information Maximization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAVTGLFP}},
  note         = {Machine review of arXiv:2506.01701}
}

read the original abstract

In this paper, we present InfoMax, a novel data pruning method, also known as coreset selection, designed to maximize the information content of selected samples while minimizing redundancy. By doing so, InfoMax enhances the overall informativeness of the coreset. The information of individual samples is measured by importance scores, which capture their influence or difficulty in model learning. To quantify redundancy, we use pairwise sample similarities, based on the premise that similar samples contribute similarly to the learning process. We formalize the coreset selection problem as a discrete quadratic programming (DQP) task, with the objective of maximizing the total information content, represented as the sum of individual sample contributions minus the redundancies introduced by similar samples within the coreset. To ensure practical scalability, we introduce an efficient gradient-based solver, complemented by sparsification techniques applied to the similarity matrix and dataset partitioning strategies. This enables InfoMax to seamlessly scale to datasets with millions of samples. Extensive experiments demonstrate the superior performance of InfoMax in various data pruning tasks, including image classification, vision-language pre-training, and instruction tuning for large language models. Code is available at https://github.com/hrtan/InfoMax.

Figures

Figures reproduced from arXiv: 2506.01701 by the authors.

**Figure 1.** Summarization of InfoMax’s performance in vision-language pre-training and image classification (both at 10% selection ratio), and instruction fine-tuning for large language models (5% selection ratio). The results show that InfoMax has demonstrated substantial progress in various scenarios, see Section 4 for details. Large-scale datasets have been pivotal in the recent breakthroughs in deep learning (Brown, 2020… view at source ↗

**Figure 2.** The coreset distributions from InfoMax, and, the score-based method (output margin [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Cases of samples with low informativeness (left) and high informativeness (right). In this paper, we introduce InfoMax, a new and effective approach for data pruning and coreset selection. Our core insight is to find a subset of samples that maximizes overall information by simultaneously considering each sample’s information contribution and the information overlap among them. First, a sample’s informativeness can… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The overall pipeline of our InfoMax. In the first two steps, we use a network to extract [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Experimental results of coreset selection on CC12M ( [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: A comparative analysis of the performance of InfoMax on instruction tuning datasets for [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Ablation study for hyperparameters in InfoMax: the dataset partition strategy (the size of [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

[3]

InfoMax achieves the best performance in the cross-model generalization ability test. Notably, when using unsupervised scores (SSP (Sorscher et al., 2022)) and unsupervised features from DINO (Oquab et al., 2023), InfoMax has better cross-model generalization ability compared to the supervised version. This also indicates the significance of using unsuper...

work page 2022
[5]

There are two main stages for InfoMax, the first one is the objective construction stage, including inferencing on all data and calculating the similarity matrix K and calculating the sample-wise score I, and the second stage is the optimizing stage, which iteratively running the solver defined in Eq. (3). It is easy to find that although the computationa...

work page 2025
[7]

Challenging big-bench tasks and whether chain-of-thought can solve them

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051,

2023
[8]

Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

arXiv
[9]

Submodularity in data subset selection and active learning

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. InInternational conference on machine learning, pp. 1954–1963. PMLR,

work page 1954
[10]

Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

arXiv
[11]

14 Published as a conference paper at ICLR 2025 BROADERIMPACT This paper presents a novel and effective data pruning algorithm to advance the deep learning area. There are some potential positive societal effects, such as helping people better understand the role of data to develop more robust deep learning systems and possibly even be used to reduce trai...

work page 2017
[15]

and Margin (Har-Peled et al., 2007)) and the type of feature (unsupervised DINO features (Oquab et al., 2023), unsupervised VQGAN features (Esser et al., 2020)). We observe that regardless of the configurations, InfoMax can achieve remarkably superior results compared to score-based approaches that only utilize score or schemes that conduct K-Median Cores...

work page 2007

Show all 24 references

[19]

The EL2N score (Paul et al.,

23 (Maharana et al., 2023)Score-based method (entropy (Cody Coleman et al., 2019))4.1 C RELATEDWORKS Score-based methods.The score-based techniques are the most popular data selection approaches. The EL2N score (Paul et al.,

2023
[20]

Influence score (Tan et al., 2023; Xia et al.,

proposed to use the distance between the sample and its corresponding cluster center as the importance score. Influence score (Tan et al., 2023; Xia et al.,

2023
[21]

Dyn-Unc He et al

can use any selection criterion, such as EL2N score (Paul et al., 2018), as a basis. Dyn-Unc He et al. (2024) proposed an efficient uncertainty-based score with awareness of training dynamics. Some related works also use sample-wise scores (Radford et al., 2021; Mahmoud et al.,

2024
[22]

to reflect the quality of multi-modality data. 18 Published as a conference paper at ICLR 2025 Diversity-based (Geometry-based) methods.Traditionally, diversity-based coreset schemes are a very classic computer science problem (Lloyd, 1982; Tan et al., 2006; Coates & Ng, 2012;...

2017
[23]

Yang et al

sets different sampling ratios for samples with different scores to enhance data coverage and balance both easy and hard samples. Yang et al. (2024) introduces reconstructing the classification boundary on the original dataset as a goal and brings it into the framework of CCS....

2024
[24]

One of the core steps of D2-Pruning is the Inverse Message Passing operation, which iteratively performs a greedy sample selection step

views data pruning as a node selection problem based on Message-Passing on a graph, where the intra-sample information is utilized as the node values on the sample graph. One of the core steps of D2-Pruning is the Inverse Message Passing operation, which iteratively performs a...

2015
[256]

Regarding data augmentation, we solely adopt RandomResizedCrop and RandomHorizontalFlip for all experiments. A.2 VISION-LANGUAGEPRETRAINING For coreset selection on the vision-language dataset CC12M (Changpinyo et al., 2021), all experiments are conducted on 2 servers with a t...

2021
[1982]

The flan collection: Designing data and methods for effective instruction tuning.arXiv preprint arXiv:2301.13688,

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning.arXiv preprint arXiv:2301.13688,

arXiv
[2007]

This is because InfoMax does not have a greedy selection process on a per-sample basis

34.3Supervised feature (K-Median diversity-based)38.9DINO feature (K-Median diversity-based) (Oquab et al., 2023)31.7VQGAN feature (K-Median diversity-based) (Esser et al., 2020)28.2InfoMax: forgetting + supervised feature 89.4InfoMax: margin + supervised feature 88.0InfoMax: ...

2023
[2015]

Estimating training data influence by tracing gradient descent.arXiv preprint arXiv:2002.08484,

Garima Pruthi, Frederick Liu, Sundararajan Mukund, and Satyen Kale. Estimating training data influence by tracing gradient descent.arXiv preprint arXiv:2002.08484,

2002 arXiv
[2018]

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

12 Published as a conference paper at ICLR 2025 Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML,

2025
[2020]

Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

arXiv
[2022]

Active learning for convolutional neural networks: A core-set approach

13 Published as a conference paper at ICLR 2025 Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach

2025
[2023]

Name Time cost (min) Stage-1 of InfoMax 14.6Stage-2 of InfoMax 1.7Overall of InfoMax 16.1 D2-Pruning (Maharana et al.,

and score-based method (Entropy (Cody Coleman et al., 2019)), for 1 million image data (the training set of ImageNet-1K (Russakovsky et al., 2015)). Name Time cost (min) Stage-1 of InfoMax 14.6Stage-2 of InfoMax 1.7Overall of InfoMax 16.1 D2-Pruning (Maharana et al.,

2019
[2024]

On coresets for k-means and k-median clustering

11 Published as a conference paper at ICLR 2025 Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing, pp. 291–300,

2025
[2048]

dog" exists, a text description like

After 1 warmup epoch, the learning rate gradually decreases from 1e-4 following the cosine strategy. Zero-shot ImageNet classification.The CLIP model has two encoders, one for text and one for images. During the zero-shot classification process, text descriptions corresponding...

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

InfoMax achieves the best performance in the cross-model generalization ability test. Notably, when using unsupervised scores (SSP (Sorscher et al., 2022)) and unsupervised features from DINO (Oquab et al., 2023), InfoMax has better cross-model generalization ability compared to the supervised version. This also indicates the significance of using unsuper...

work page 2022

[2] [5]

There are two main stages for InfoMax, the first one is the objective construction stage, including inferencing on all data and calculating the similarity matrix K and calculating the sample-wise score I, and the second stage is the optimizing stage, which iteratively running the solver defined in Eq. (3). It is easy to find that although the computationa...

work page 2025

[3] [7]

Challenging big-bench tasks and whether chain-of-thought can solve them

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051,

2023

[4] [8]

Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

arXiv

[5] [9]

Submodularity in data subset selection and active learning

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. InInternational conference on machine learning, pp. 1954–1963. PMLR,

work page 1954

[6] [10]

Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

arXiv

[7] [11]

14 Published as a conference paper at ICLR 2025 BROADERIMPACT This paper presents a novel and effective data pruning algorithm to advance the deep learning area. There are some potential positive societal effects, such as helping people better understand the role of data to develop more robust deep learning systems and possibly even be used to reduce trai...

work page 2017

[8] [15]

and Margin (Har-Peled et al., 2007)) and the type of feature (unsupervised DINO features (Oquab et al., 2023), unsupervised VQGAN features (Esser et al., 2020)). We observe that regardless of the configurations, InfoMax can achieve remarkably superior results compared to score-based approaches that only utilize score or schemes that conduct K-Median Cores...

work page 2007