Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
The role of permutation invariance in linear mode connectivity of neural networks.arXiv preprint arXiv:2110.06296
11 Pith papers cite this work. Polarity classification is still indexing.
11
Pith papers citing it
abstract
In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
In grokking modular arithmetic, weight direction portably carries circuit identity across independent runs while weight norm only sets susceptibility to overwrite and a weak delay effect.
Tiered Language Models use a secret key to induce an alternative computation graph over shared weights, enabling private capabilities in the keyed mode while the public mode shows none.
Fisher-MoE prunes sparse intermediate dimensions in MoE FFNs ranked by Fisher importance, delivering 50% compression that preserves capability while cutting memory ~45% and raising throughput 21%.
Optimal layer-patching order for boomerang distillation is a shortest path on a KL-weighted Boolean lattice; greedy KLPatch and simple sequential orders often yield near-optimal interpolations.
MCWC aligns permutation-symmetric blocks across layers to enable sequential prediction and residual entropy coding, improving rate-accuracy tradeoffs versus quantization and prior codecs on language and vision models.
ODE-M formulates continual model merging as a barrier-aware ODE trajectory in parameter space, using first-order feedback and a utility-aware schedule to balance retained knowledge and new task performance.
A bidirectional optimization method using parameterized transformations enables near-zero loss barriers for linear mode connectivity in medium-scale language models and small barriers in billion-parameter transformers.
DiMS is a physics-inspired dynamical sampler guaranteed to exactly sample reparameterization-invariant minimum level sets in neural network loss landscapes.
Representations learned by large AI models are converging toward a shared statistical model of reality.
Humans and VLMs diverge in VQA responses on driving footage, with human answers consistent across origins and no strong geography modulation observed, likely due to high OOD nature.
citing papers explorer
-
Editing Models with Task Arithmetic
Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
-
Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking
In grokking modular arithmetic, weight direction portably carries circuit identity across independent runs while weight norm only sets susceptibility to overwrite and a weak delay effect.
-
Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs
Tiered Language Models use a secret key to induce an alternative computation graph over shared weights, enabling private capabilities in the keyed mode while the public mode shows none.
-
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Fisher-MoE prunes sparse intermediate dimensions in MoE FFNs ranked by Fisher importance, delivering 50% compression that preserves capability while cutting memory ~45% and raising throughput 21%.
-
Understanding Layer Patching in Model Size Interpolation
Optimal layer-patching order for boomerang distillation is a shortest path on a KL-weighted Boolean lattice; greedy KLPatch and simple sequential orders often yield near-optimal interpolations.
-
Motion-Compensated Weight Compression
MCWC aligns permutation-symmetric blocks across layers to enable sequential prediction and residual entropy coding, improving rate-accuracy tradeoffs versus quantization and prior codecs on language and vision models.
-
Unlocking the Potential of Continual Model Merging: An ODE Perspective
ODE-M formulates continual model merging as a barrier-aware ODE trajectory in parameter space, using first-order feedback and a utility-aware schedule to balance retained knowledge and new task performance.
-
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
A bidirectional optimization method using parameterized transformations enables near-zero loss barriers for linear mode connectivity in medium-scale language models and small barriers in billion-parameter transformers.
-
Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics
DiMS is a physics-inspired dynamical sampler guaranteed to exactly sample reparameterization-invariant minimum level sets in neural network loss landscapes.
-
The Platonic Representation Hypothesis
Representations learned by large AI models are converging toward a shared statistical model of reality.
-
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
Humans and VLMs diverge in VQA responses on driving footage, with human answers consistent across origins and no strong geography modulation observed, likely due to high OOD nature.