Pith. sign in

REVIEW 4 cited by

Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.05424 v1 pith:ZNS2TPAS submitted 2023-10-09 cs.CL

Sangmin Bae , Jongwoo Ko , Hwanjun Song , Se-Young Yun This is my paper
classification cs.CL
keywords frameworkdecodingearly-exitingmodelsparalleltokenadaptiveautoregressive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

To tackle the high inference latency exhibited by autoregressive language models, previous studies have proposed an early-exiting framework that allocates adaptive computation paths for each token based on the complexity of generating the subsequent token. However, we observed several shortcomings, including performance degradation caused by a state copying mechanism or numerous exit paths, and sensitivity to exit confidence thresholds. Consequently, we propose a Fast and Robust Early-Exiting (FREE) framework, which incorporates a shallow-deep module and a synchronized parallel decoding. Our framework enables faster inference by synchronizing the decoding process of the current token with previously stacked early-exited tokens. Furthermore, as parallel decoding allows us to observe predictions from both shallow and deep models, we present a novel adaptive threshold estimator that exploits a Beta mixture model to determine suitable confidence thresholds. We empirically demonstrated the superiority of our proposed framework on extensive generation tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving

    cs.DC 2026-04 unverdicted novelty 7.0 of 10

    FASER delivers up to 53% higher throughput and 1.92x lower latency in dynamic LLM serving by adjusting speculative lengths per request, early pruning of rejects, and overlapping draft/verification phases via frontiers.

  2. LLM-assisted Agentic Edge Intelligence Framework

    cs.DC 2026-03 unverdicted novelty 5.0 of 10

    LEI framework uses a cloud LLM to dynamically create and update tailored lightweight programs for heterogeneous edge devices, shown on four sensor datasets to maintain low CPU and memory use while adapting to changes.

  3. LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation

    cs.CL 2025-07 unverdicted novelty 5.0 of 10

    LogitSpec accelerates retrieval-based speculative decoding by speculating the next-next token from the last logit and retrieving relevant references for both next and next-next tokens, reporting up to 2.61x speedup an...

  4. Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes

    cs.CL 2026-08 conditional novelty 4.0 of 10

    Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.

Pith tools