pith. sign in

arxiv: 1606.01305 · v4 · pith:QRG3KU34new · submitted 2016-06-03 · 💻 cs.NE · cs.CL· cs.LG

Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

David Krueger , Tegan Maharaj , J\'anos Kram\'ar , Mohammad Pezeshki , Nicolas Ballas , Nan Rosemary Ke , Anirudh Goyal , Yoshua Bengio
show 2 more authors
Aaron Courville Chris Pal
This is my paper
classification 💻 cs.NE cs.CLcs.LG
keywords zoneouthiddeninformationpreservingregularizingresultsrnnsunits
0
0 comments X
read the original abstract

We propose zoneout, a novel method for regularizing RNNs. At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization. But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks. We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Pointer Sentinel Mixture Models

    cs.CL 2016-09 conditional novelty 7.0

    Pointer sentinel-LSTM mixes context copying with softmax prediction to reach 70.9 perplexity on Penn Treebank using fewer parameters than standard LSTMs.

  2. One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models

    cs.LG 2026-04 unverdicted novelty 6.0

    Denoising Recursion Models train multi-step noise reversal in looped transformers and outperform the prior Tiny Recursion Model on ARC-AGI.

  3. DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

    cs.CL 2019-07 unverdicted novelty 6.0

    DropAttention regularizes attention weights in fully-connected self-attention networks to reduce overfitting and improve performance.

  4. R-Transformer: Recurrent Neural Network Enhanced Transformer

    cs.LG 2019-07 unverdicted novelty 6.0

    R-Transformer integrates RNNs with multi-head attention to model local and global sequence dependencies without position embeddings and reports large-margin gains over prior methods on diverse tasks.

  5. ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network

    cs.LG 2019-06 unverdicted novelty 5.0

    ARMIN introduces auto-addressing via hidden states and a novel RNN cell to produce a lighter recurrent memory network with lower overhead than existing MANNs or vanilla LSTMs.

  6. A Comprehensive Overview of Large Language Models

    cs.CL 2023-07 unverdicted novelty 2.0

    A survey paper providing an overview of Large Language Models, their background, and recent advances in the field.