Neural Network Diffusion
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:A42OAK6Frecord.jsonopen to challenge →
read the original abstract
Diffusion models have achieved remarkable success in image and video generation. In this work, we demonstrate that diffusion models can also \textit{generate high-performing neural network parameters}. Our approach is simple, utilizing an autoencoder and a diffusion model. The autoencoder extracts latent representations of a subset of the trained neural network parameters. Next, a diffusion model is trained to synthesize these latent representations from random noise. This model then generates new representations, which are passed through the autoencoder's decoder to produce new subsets of high-performing network parameters. Across various architectures and datasets, our approach consistently generates models with comparable or improved performance over trained networks, with minimal additional cost. Notably, we empirically find that the generated models are not memorizing the trained ones. Our results encourage more exploration into the versatile use of diffusion models. Our code is available \href{https://github.com/NUS-HPC-AI-Lab/Neural-Network-Diffusion}{here}.
This paper has not been read by Pith yet.
Forward citations
Cited by 6 Pith papers
-
Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork
Hypernetwork generates model parameters from one perturbed low-dimensional private dataset embedding, yielding higher utility than DP-SGD under fixed privacy budget in synthetic theory and lower FID in LoRA diffusion ...
-
Good Agentic Friends Do Not Just Give Verbal Advice: They Can Update Your Weights
TFlow enables multi-agent LLMs to collaborate via transient low-rank LoRA perturbations derived from sender activations, yielding up to 8.5 accuracy gains and 83% token reduction versus text-based baselines on Qwen3-4...
-
Generative Adaptation of Dynamics to Environmental Shifts via Weight-space Diffusion
DynaDiff uses weight-graph diffusion with a functional consistency loss and dynamics-informed prompting to generate adapted predictors, reporting 10.78% average accuracy gains over baselines while amortizing adaptatio...
-
Fast Kernel-Space Diffusion for Remote Sensing Pansharpening
KSDiff generates convolutional kernels in kernel space using low-rank core tensor and factor generators with multi-head attention for fast, high-quality pansharpening.
-
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
A bidirectional optimization method using parameterized transformations enables near-zero loss barriers for linear mode connectivity in medium-scale language models and small barriers in billion-parameter transformers.
-
Diffusion Models are Evolutionary Algorithms
Diffusion models are evolutionary algorithms via a denoising-evolution equivalence, yielding Diffusion Evolution that outperforms mainstream EAs on multi-optima tasks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.