Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
Essentially No Barriers in Neural Network Energy Landscape
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Permutation symmetries generate permutation saddles and equal-loss valleys linking equivalent global minima, yielding a lower bound on symmetry-induced critical points.
NEB-adapted ravine ensembles for QNNs classifying concentratable entanglement outperform naive methods when local-prediction variability is high and reduce costs, with ravines persisting under depth and qubit scaling.
Local low-rank task-gradient structures exist in weights and activations but are non-stationary, with initial recovery updates forming a basis capturing 77% of LoRA displacement and parameter steps aligning 0.58 cosine with CAA steering vectors.
citing papers explorer
-
Editing Models with Task Arithmetic
Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
-
Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
Permutation symmetries generate permutation saddles and equal-loss valleys linking equivalent global minima, yielding a lower bound on symmetry-induced critical points.
-
Ravines in quantum cost landscapes: opportunities for improved VQA predictions
NEB-adapted ravine ensembles for QNNs classifying concentratable entanglement outperform naive methods when local-prediction variability is high and reduce costs, with ravines persisting under depth and qubit scaling.
-
Recoverable but Not Stationary:Local Linear Structures in Weights and Activations
Local low-rank task-gradient structures exist in weights and activations but are non-stationary, with initial recovery updates forming a basis capturing 77% of LoRA displacement and parameter steps aligning 0.58 cosine with CAA steering vectors.