REVIEW 33 cited by
An Empirical Study of Example Forgetting during Deep Neural Network Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
An Empirical Study of Example Forgetting during Deep Neural Network Learning
read the original abstract
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.
Forward citations
Cited by 33 Pith papers
-
Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining
During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.
-
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
MergeProbe forecasts LoRA adapter mergeability from first-few-percent training signals and outperforms interference-aware baselines on retention while adding low overhead on a five-domain benchmark.
-
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
Shared chat-template tokens piggyback narrow finetuning behaviors onto out-of-domain queries; regularizing their KV states (TReFT) reduces emergent misalignment and other off-topic generalization.
-
Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning
Three last-layer coefficients derived from gradient interference sources predict the forgetting rank order of past classes in rehearsal-based CIL, with self-induced interference as the strongest predictor.
-
Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training
PODS is a plug-and-play oscillatory data-volume scheduler that alternates low-ratio regularization phases with high-ratio recovery phases to improve data selection efficiency across training tasks.
-
The Benefits of Temporal Correlations: SGD Learns k-Juntas from Random Walks Efficiently
Temporal correlations from lazy random walks enable efficient SGD learning of k-juntas via temporal-difference loss on ReLU networks, achieving linear sample complexity in d.
-
Eliciting Latent Predictions from Transformers with the Tuned Lens
Training per-layer affine probes on frozen transformers yields more reliable latent predictions than the logit lens and enables detection of malicious inputs from prediction trajectories.
-
Reward Valuation in Vision Language Models: Causal Mechanisms Underlying Anhedonia
Targeted perturbation of reward-anticipatory units in VLMs induces anhedonia-like effort avoidance and clinical-scale score drops without impairing baseline task competence.
-
Dataset Distillation Based on Saliency-Driven Prototype Alignment
Saliency-guided and confidence-refined latent prototypes improve diffusion-based dataset distillation accuracy across multiple benchmarks without fine-tuning the diffusion model.
-
Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
RAHA applies rank-aware hyperbolic alignment to vision-language dataset distillation by enforcing geodesic alignment in the shared low-rank range and regularizing the residual subspace for improved transfer.
-
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
The Piggyback Hypothesis attributes emergent misalignment to chat-template tokens piggybacking finetuned behavior; Token-Regularized Finetuning (TReFT) mitigates it by regularizing prefix token representations.
-
Trajectory-Based Difficulty Scoring for Reliable Learning on Tabular Data
TDS uses per-tree prediction trajectories to derive instance difficulty scores that rank errors better than prior hardness measures and improve active learning, selective prediction, and Mondrian conformal prediction ...
-
Multimodal Distribution Matching for Vision-Language Dataset Distillation
MDM distills vision-language datasets via joint embedding clustering, weight-space model interpolation, and geometry-aware distribution matching on the unit hypersphere.
-
LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection
LiBaGS scores and selects synthetic data near decision boundaries using proximity, uncertainty, density, and validity, with boundary-gap allocation and marginal stopping to improve training accuracy.
-
Let the Target Select for Itself: Data Selection via Target-Aligned Paths
Target-aligned data selection via normalized endpoint loss drop on a validation-induced reference path achieves competitive performance with reduced computational overhead.
-
Leveraging Data Symmetries to Select an Optimal Subset of Training Data under Label Noise
Exploiting data symmetries boosts k-NN to select near-optimal low-noise subsets from noisy datasets, approaching Bayes-optimal performance in high dimensions, with learned representations aiding partial symmetry knowledge.
-
Not All Forgetting Is Equal: Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers
Fine-tuning causes architecture-specific forgetting of individual samples that is stochastic across seeds, with ViTs showing more predictable decay than CNNs and class-level patterns that are semantically consistent.
-
Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment
AlignPrune uses a Dynamic Alignment Score from loss trajectories to identify noisy samples more accurately than per-sample loss, improving pruning accuracy by up to 6.3% on noisy benchmarks.
-
OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation
Dynamic hierarchical data pruning improves NDCG@10 and Recall@20 for dense retrievers while reaching full performance in half the iterations.
-
Data Agent: Learning to Select Data via End-to-End Dynamic Optimization
Data Agent learns a co-evolving sample selection policy end-to-end that accelerates training by over 50% on ImageNet-1k and MMLU with no performance loss.
-
BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuning
Fine-tuning a benign teacher on a weak trigger at a 100x-reduced learning rate is sufficient to make the backdoor survive knowledge distillation into student models.
-
Surprisingly High Redundancy in Electronic Structure Data Across Materials Explained by Low Intrinsic Dimensionality
Electronic structure datasets across materials show high redundancy from low intrinsic dimensionality, allowing pruning to 1/100th size with preserved chemical accuracy.
-
EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training
EPS uses DCT features to cluster patches by spatial-temporal complexity and adaptively samples from the highest cluster, cutting training patches by 75-91.69% and speeding sampling up to 82.1x versus EMT while claimin...
-
Demystifying CLIP Data
MetaCLIP curates balanced 400M-pair subsets from CommonCrawl that outperform CLIP data, reaching 70.8% zero-shot ImageNet accuracy on ViT-B versus CLIP's 68.3%.
-
Dataset Distillation Based on Saliency-Driven Prototype Alignment
Saliency-guided latent prototypes plus confidence-based hard-prototype refinement improve diffusion-based dataset distillation accuracy on ImageNet subsets, CIFAR, and ImageNet-1K without fine-tuning the generative backbone.
-
Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?
Large-scale standardized benchmarks show state-of-the-art dataset distillation methods do not outperform coreset selection on ImageNet-scale data and have substantially higher construction costs.
-
OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework
OrderDP is a plug-and-play data pruning method that selects a random subset then top-q samples to guarantee unbiased surrogate-loss training with convergence analysis and over 40% training cost reduction on CIFAR and ...
-
LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection
LiBaGS is a lightweight method that picks synthetic data near decision boundaries while checking density and validity to improve training accuracy over standard oversampling or uncertainty sampling.
-
Labeled TrustSet Guided: Batch Active Learning with Reinforcement Learning
BRAL-T uses TrustSet-guided reinforcement learning for batch active learning and reports state-of-the-art results on 10 image classification benchmarks plus 2 fine-tuning tasks.
-
Improving Translation Quality by Selecting Better Data for LLM Fine-Tuning: A Comparative Analysis
Fine-tuning on data selected by COMET-Kiwi sometimes improves Japanese-English translation, but the paper's claim that semantic selectors consistently win is contradicted by its own results.
-
LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training
LiLAW learns to weight samples as easy, moderate or hard using three global scalars updated by one gradient step on a validation batch to improve noisy training performance.
-
Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
Selecting per-class medoids (samples with lowest average L2 distance to all same-class samples in teacher feature space) consistently outperforms random, herding, and k-center Greedy baselines for few-shot knowledge d...
-
Using Explainability as a Training-Time Reliability Signal for Efficient ECG Classification
ERTS applies explanation quality from Grad-CAM to select reliable samples in progressive data dropout for ECG classification, reporting higher macro-F1 and lower training cost across three datasets and multiple architectures.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.