Pith. sign in

REVIEW 11 cited by

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.16710 v4 pith:KBVO7DIZ submitted 2024-04-25 cs.CL cs.AIcs.LG

Mostafa Elhoushi , Akshat Shrivastava , Diana Liskovich , Basil Hosmer , Bram Wasti , Liangzhen Lai , Anas Mahmoud , Bilge Acun
show 5 more authors
Saurabh Agarwal Ahmed Roman Ahmed A Aly Beidi Chen Carole-Jean Wu
This is my paper
classification cs.CLcs.AIcs.LG
keywords layersexitdecodingearlyinferencedropoutlayerskipmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and checkpoints at https://github.com/facebookresearch/LayerSkip.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.

  2. Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

    cs.AI 2026-07 conditional novelty 6.0 of 10

    In a depth-recurrent transformer, each token converges to a fixed point at its own rate; a parameter-free early-exit rule reads this and matches depth-8 quality at 4.94 average loops, beating a learned router.

  3. HiSpec: Hierarchical Speculative Decoding for LLMs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    HiSpec uses early-exit layers to verify draft tokens midway through the model, reporting 1.28×-2.01× faster decoding over baseline speculative decoding, but the accuracy claim is not empirically tested.

  4. DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A learned router sends about 90% of tokens through a linear projection path instead of quadratic attention, keeps the MLP for every token, and matches dense Transformer accuracy at roughly 85% of the FLOPs.

  5. Towards Distributed Neural Architectures

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A general routing framework lets each token take its own path through a network, matching dense baselines while enabling learned compute savings and interpretable path specialization.

  6. FREE: Fast and Robust Vision Language Models with Early Exits

    cs.LG 2025-06 conditional novelty 6.0 of 10

    An adversarial early-exit method for frozen-backbone vision language models that reuses the final classifier and reports 1.5x inference speedup with comparable accuracy.

  7. POSS: Position Specialist Generates Better Draft for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Using position-specialized draft layers instead of one single draft model improves later-token acceptance in speculative decoding, yielding modest speedups on Llama-3-8B and Llama-2-13B.

  8. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...

  9. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  10. DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policies

    cs.CL 2025-05 conditional novelty 5.0 of 10

    DASH trains a token-level policy to skip, quantize, or fully run each Transformer layer, reporting 1.33x-2.0x compute reduction with smaller accuracy loss than prior skipping methods.

  11. Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

    cs.CV 2025-05 conditional novelty 5.0 of 10

    FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.

Pith tools