Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
Visualizing the Loss Landscape of Neural Nets
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
abstract
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
NEB-adapted ravine ensembles for QNNs classifying concentratable entanglement outperform naive methods when local-prediction variability is high and reduce costs, with ravines persisting under depth and qubit scaling.
Rescaled ASGD recovers convergence to the true global objective by rescaling worker stepsizes proportional to computation times, matching the known time lower bound in the leading term under non-convex smoothness and bounded heterogeneity.
SAM solves a min-max problem to locate flat low-loss regions, improving generalization on CIFAR, ImageNet and label-noise tasks.
A multiscale optimization method using explicit protein backbone geometry reconstructs atomic models from cryo-EM data, showing improved RMSD and TM scores on three simulated datasets.
A linear stability analysis introduces data coherence to explain why SGD and SAM prefer stable and simple minima in two-layer ReLU networks.
Adversarial training enhances robustness of jet tagging classifiers while preserving performance, with loss surface geometry providing insights into correlations and vulnerability.
GNC convolves stochastic gradient noise to smooth sharp minima in large-batch SGD, outperforming isotropic noise for better generalization in distributed deep learning.
A conditional Wasserstein GAN generates complete kinematic event summaries for IBD-CC, NC, and NuEElastic electron neutrino interactions that match GENIE distributions in 1D marginals and correlations.
A comprehensive review of deep learning techniques for computational mechanics, including LSTM for constitutive modeling, PINNs for PDE solving, optimizers, and kernel methods.
citing papers explorer
-
Multiscale reconstruction of protein conformations from cryo-EM images
A multiscale optimization method using explicit protein backbone geometry reconstructs atomic models from cryo-EM data, showing improved RMSD and TM scores on three simulated datasets.