REVIEW 9 cited by
Fantastic Generalization Measures and Where to Find Them
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.
Forward citations
Cited by 9 Pith papers
-
Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learning
SAGE, an optimizer combining spectral polar-factor perturbation with gradient-agreement-scaled noise, reports 78.9% average on DomainBed.
-
One task to rule them all: A closer look at traffic classification generalizability
Traffic classifiers that seem near-perfect on their own datasets fall to 30-40% accuracy on another network's same-task data, and a 1-Nearest Neighbor baseline is competitive.
-
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
SAM's largest Hessian eigenvalue is bounded by the cube root of bGamma/(2*rho*eta^2), so larger radius, smaller batch, or larger learning rate restrict linearly stable minima to flatter regions.
-
Decentralized SGD with Controlled Disagreement Finds Flatter Minima
Keeping consensus errors alive in decentralized SGD via a learning-rate-scaled mixing term improves test accuracy and flatter minima over both DSGD and synchronous SGD.
-
How Far Are We from True Unlearnability?
Current unlearnable examples fail under multi-task training, and the proposed SAL and UD metrics quantify how far each method is from true unlearnability.
-
DHEvo: Data-Algorithm Based Heuristic Evolution for Generalizable MILP Solving
DHEvo co-evolves MILP training instances and diving heuristics, improving generalization over existing LLM-based heuristic generation methods.
-
Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
GEAR-SAM re-allocates SAM's fixed perturbation radius across network blocks in proportion to an EMA of squared block-gradient norms, improving generalization on CIFAR, transfer, and label-noise benchmarks.
-
VASSO: Variance Suppression for Sharpness-Aware Minimization
VASSO replaces SAM's minibatch gradient with an exponential moving average of past gradients when computing the adversarial perturbation, improving generalization across vision and language tasks.
-
Automatic Stability and Recovery for Neural Network Training
A validation-loss-triggered rollback controller whose "safety guarantees" restate its own accept/reject rule.
Discussion (0). Sign in to comment.