Linear Mode Connectivity and the Lottery Ticket Hypothesis
read the original abstract
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Measuring Alignment-Induced Activation Shifts Correctly: A Template-Controlled Difference-in-Differences Protocol
Introduces a template-controlled difference-in-differences protocol that corrects chat-template confounding when measuring alignment-induced activation shifts in LLMs and recovers the refusal direction with higher fidelity.
-
Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior
Data-similarity and data-influence produce significantly overlapping rankings of training documents for LLM outputs, with asymmetry allowing a favorable cost-accuracy trade-off.
-
Surrogate Neural Architecture Codesign Package (SNAC-Pack)
SNAC-Pack automates hardware-aware neural architecture codesign for FPGAs via surrogate-based multi-objective search, QAT/pruning compression, and hls4ml synthesis, yielding compact models with reduced resources on je...
-
Surrogate Neural Architecture Codesign Package (SNAC-Pack)
SNAC-Pack is a new framework for hardware-aware neural architecture codesign that uses surrogate models, NSGA-II search, quantization-aware training, and hls4ml synthesis to produce compact FPGA-deployable models.
-
Explaining Machine Learning and Memorization with Statistical Mechanics
Thesis uses statistical mechanics to study DAM and RBM models for understanding memorization, low-dimensional learning, and adversarial robustness in neural networks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.