Pith. sign in

REVIEW 11 cited by

Reducing Transformer Depth on Demand with Structured Dropout

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.11556 v1 pith:M2MKVMQA submitted 2019-09-25 cs.LG cs.CLstat.ML

Angela Fan , Edouard Grave , Armand Joulin This is my paper
classification cs.LGcs.CLstat.ML
keywords languageansweringapproachdepthdropoutlargemachinemodeling
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality compared to training from scratch or using distillation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 273 citations worldwide. Full citation record

  1. VISP: Volatility Informed Stochastic Projection for Adaptive Regularization

    cs.LG 2025-09 reject novelty 6.0 of 10

    VISP applies gradient-volatility-scaled stochastic projection to activations and reports improved test error on three image benchmarks, but without error bars or code.

  2. Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Study

    cs.CL 2026-08 conditional novelty 5.0 of 10

    Pruning multilingual NMT vocabularies to corpus-relevant tokens plus fine-tuning cuts memory by about 60% and matches or beats a dedicated English-Arabic model on COMET and TER.

  3. Harnessing Input-Adaptive Inference for Efficient VLN

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A three-part input-adaptive inference method (view masking, adaptive early exit, view caching) cuts VLN computation roughly in half on seven benchmarks with moderate success-rate loss.

  4. Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    A systematic review of 55 papers finds explainability for multimodal attention-based models is dominated by attention-weight visualizations, while evaluation remains mostly qualitative and non-standardized.

  5. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  6. AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    An iterative Transformer that reuses its middle layers with an extra residual connection matches a standard Transformer after one pass and improves perplexity and zero-shot accuracy when run for more passes at test time.

  7. Towards Universal & Efficient Model Compression via Exponential Torque Pruning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Exponential Torque Pruning applies an exponential penalty on network modules based on their distance from a pivot, improving compression accuracy trade-offs over linear Torque pruning.

  8. Learning to Skip the Middle Layers of Transformers

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A gated 'skip the middle' Transformer fails to beat fewer-layer dense baselines on the cross-entropy/FLOPs trade-off at 12 layers and 10B tokens.

  9. Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs

    cs.LG 2025-07 reject novelty 4.0 of 10

    Pretrained LLM layers can be skipped/repeated per input to build custom paths, but the search uses ground-truth answers, so the accuracy gains are fitted, not predicted.

  10. Position: The Future of Bayesian Prediction Is Prior-Fitted

    cs.LG 2025-05 conditional novelty 4.0 of 10

    PFNs, which amortize Bayesian inference by training on datasets sampled from a prior, are likely to supersede MCMC and variational inference for most prediction tasks, the authors argue.

  11. AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A vision transformer reduces self-attention complexity from O(n^2) to O(mn) by using m learnable anchor tokens and a two-step Markov transition between anchors and tokens.

Pith tools