REVIEW 7 cited by
Dataset Distillation with Convexified Implicit Gradients
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Dataset Distillation with Convexified Implicit Gradients
read the original abstract
We propose a new dataset distillation algorithm using reparameterization and convexification of implicit gradients (RCIG), that substantially improves the state-of-the-art. To this end, we first formulate dataset distillation as a bi-level optimization problem. Then, we show how implicit gradients can be effectively used to compute meta-gradient updates. We further equip the algorithm with a convexified approximation that corresponds to learning on top of a frozen finite-width neural tangent kernel. Finally, we improve bias in implicit gradients by parameterizing the neural network to enable analytical computation of final-layer parameters given the body parameters. RCIG establishes the new state-of-the-art on a diverse series of dataset distillation tasks. Notably, with one image per class, on resized ImageNet, RCIG sees on average a 108\% improvement over the previous state-of-the-art distillation algorithm. Similarly, we observed a 66\% gain over SOTA on Tiny-ImageNet and 37\% on CIFAR-100.
Forward citations
Cited by 7 Pith papers
-
Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets
Introduces SGR and TIAT for robust dataset distillation that suppresses noise while preserving knowledge under noisy supervision.
-
Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
RAHA applies rank-aware hyperbolic alignment to vision-language dataset distillation by enforcing geodesic alignment in the shared low-rank range and regularizing the residual subspace for improved transfer.
-
Multimodal Distribution Matching for Vision-Language Dataset Distillation
MDM distills vision-language datasets via joint embedding clustering, weight-space model interpolation, and geometry-aware distribution matching on the unit hypersphere.
-
DIVER:Diving Deeper into Distilled Data via Expressive Semantic Recovery
DIVER is a dual-stage distillation method using diffusion models to enhance semantic preservation and cross-architecture generalization in dataset distillation.
-
DIVER:Diving Deeper into Distilled Data via Expressive Semantic Recovery
DIVER applies a pre-trained diffusion model in a dual-stage process of semantic inheritance, guidance, and fusion to improve semantic expression and cross-architecture generalization in dataset distillation.
-
Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration
FedHD performs federated distillation for whole slide images by generating one synthetic feature set per real slide via Gaussian-mixture alignment and adding them via curriculum integration, outperforming prior federa...
-
Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration
FedHD is a federated learning framework for whole slide images that distills one-to-one synthetic features aligned via Gaussian mixtures and progressively integrates cross-site features through curriculum learning to ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.