Pith. sign in

REVIEW 21 cited by

Reducing Transformer Depth on Demand with Structured Dropout

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.11556 v1 pith:M2MKVMQA submitted 2019-09-25 cs.LG cs.CLstat.ML

Angela Fan , Edouard Grave , Armand Joulin This is my paper · ORCID
classification cs.LGcs.CLstat.ML
keywords languageansweringapproachdepthdropoutlargemachinemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality compared to training from scratch or using distillation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 273 citations worldwide. Full citation record

  1. VISP: Volatility Informed Stochastic Projection for Adaptive Regularization

    cs.LG 2025-09 reject novelty 6.0 of 10

    VISP applies gradient-volatility-scaled stochastic projection to activations and reports improved test error on three image benchmarks, but without error bars or code.

  2. PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    PGB prunes BERT in a single pass by permuting weight matrices into block-diagonal groups of important weights and removing all other weights, matching or beating iterative baselines on GLUE and SQuAD.

  3. BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts

    cs.LG 2025-02 conditional novelty 6.0 of 10

    BEEM aggregates weighted confidence from consistent neighboring exit classifiers, resetting on disagreement, and sets thresholds from validation error rates to accelerate early-exit inference.

  4. Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Study

    cs.CL 2026-08 conditional novelty 5.0 of 10

    Pruning multilingual NMT vocabularies to corpus-relevant tokens plus fine-tuning cuts memory by about 60% and matches or beats a dedicated English-Arabic model on COMET and TER.

  5. Harnessing Input-Adaptive Inference for Efficient VLN

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A three-part input-adaptive inference method (view masking, adaptive early exit, view caching) cuts VLN computation roughly in half on seven benchmarks with moderate success-rate loss.

  6. Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    A systematic review of 55 papers finds explainability for multimodal attention-based models is dominated by attention-weight visualizations, while evaluation remains mostly qualitative and non-standardized.

  7. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  8. AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    An iterative Transformer that reuses its middle layers with an extra residual connection matches a standard Transformer after one pass and improves perplexity and zero-shot accuracy when run for more passes at test time.

  9. Towards Universal & Efficient Model Compression via Exponential Torque Pruning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Exponential Torque Pruning applies an exponential penalty on network modules based on their distance from a pivot, improving compression accuracy trade-offs over linear Torque pruning.

  10. Learning to Skip the Middle Layers of Transformers

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A gated 'skip the middle' Transformer fails to beat fewer-layer dense baselines on the cross-entropy/FLOPs trade-off at 12 layers and 10B tokens.

  11. Reasoning Bias of Next Token Prediction Training

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Training on all tokens (next token prediction) beats training only on answer tokens (critical token prediction) on small-scale reasoning benchmarks, an effect the authors attribute to noise-induced regularization.

  12. LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers

    cs.LG 2024-12 conditional novelty 5.0 of 10

    LazyDiT learns small gates that decide when to reuse cached layer outputs, cutting diffusion transformer compute by up to half while matching or beating DDIM quality.

  13. ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A domain-adversarial objective over random halves of a single training set is proposed as a general-purpose neural network regularizer, with mixed empirical support across benchmarks.

  14. Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs

    cs.LG 2025-07 reject novelty 4.0 of 10

    Pretrained LLM layers can be skipped/repeated per input to build custom paths, but the search uses ground-truth answers, so the accuracy gains are fitted, not predicted.

  15. Position: The Future of Bayesian Prediction Is Prior-Fitted

    cs.LG 2025-05 conditional novelty 4.0 of 10

    PFNs, which amortize Bayesian inference by training on datasets sampled from a prior, are likely to supersede MCMC and variational inference for most prediction tasks, the authors argue.

  16. AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A vision transformer reduces self-attention complexity from O(n^2) to O(mn) by using m learnable anchor tokens and a two-step Markov transition between anchors and tokens.

  17. Position: Enough of Scaling LLMs! Lets Focus on Downscaling

    cs.CL 2025-05 conditional novelty 4.0 of 10

    The paper advocates for downscaling LLMs, derives that training CO2 scales linearly with parameters and data, and proposes a condition under which ensembles of pruned small models beat the original at equal compute.

  18. Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning

    cs.CV 2025-04 reject novelty 4.0 of 10

    LVTP prunes transformer tokens during semantic segmentation using multi-scale Tsallis entropy with Sobel edge guidance, reporting 20-46% FLOP reduction with a few mIoU points lost and no retraining.

  19. A Survey of Early Exit Deep Neural Networks in NLP

    cs.LG 2025-01 conditional novelty 3.0 of 10

    A review of early exit deep neural network methods in NLP that has no new experiments but organizes the existing literature.

  20. On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages

    cs.CL 2025-01 reject novelty 3.0 of 10

    Layer-pruned MahaBERT-v2 and Google-Muril models roughly match full models on Marathi headline and paragraph classification but lose ground on document classification, and they do not always beat same-size scratch-tra...

  21. Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition

    cs.SD 2024-11 conditional novelty 3.0 of 10

    Inserting a linear layer before a Transformer feed-forward layer during training and merging it at inference improves lightweight speech emotion recognition models with no added inference cost.

Pith tools