REVIEW 4 cited by
Symmetry-invariant optimization in deep networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent works have highlighted scale invariance or symmetry that is present in the weight space of a typical deep network and the adverse effect that it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show that these and other commonly used deep networks, such as those which use a max-pooling and sub-sampling layer, possess more complex forms of symmetry arising from scaling based reparameterization of the network weights. We then propose two symmetry-invariant gradient based weight updates for stochastic gradient descent based learning. Our empirical evidence based on the MNIST dataset shows that these updates improve the test performance without sacrificing the computational efficiency of the weight updates. We also show the results of training with one of the proposed weight updates on an image segmentation problem.
Forward citations
Cited by 4 Pith papers
-
Path-conditioned training: a principled way to rescale ReLU neural networks
Rescaling ReLU weights at initialization with PathCond aligns the path kernel with the identity and trains to the same accuracy in up to 1.5× fewer epochs.
-
TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networks
Pruning weights and teleporting activations before proof generation cuts ZK-SNARK prover memory by up to 67% and proof time by up to 54% on vision models at about 1% accuracy cost.
-
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Transformer attention layers admit continuous rotation symmetries, and aligning a source model's attention weights by the optimal rotation before weight averaging improves model fusion accuracy.
-
Improving Learning to Optimize Using Parameter Symmetries
A new lemma shows the symmetry part of Newton's direction increases gradient norm for convex objectives, but the paper's own experiments show teleportation-augmented L2O underperforms vanilla L2O, with momentum yieldi...
Discussion (0). Continue with ORCID to comment.