Central-difference random gradient estimation (CD-RGE) trains RNNs up to 1.1B parameters without backpropagation through time, matching or beating BPTT on several small tasks while using less GPU memory.
Suraj Srinivas Malladi, Xiang Wei, Josip Djolonga, and Dale Schuurmans
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization
Central-difference random gradient estimation (CD-RGE) trains RNNs up to 1.1B parameters without backpropagation through time, matching or beating BPTT on several small tasks while using less GPU memory.