REVIEW 14 cited by
A Thorough Examination of Decoding Methods in the Era of LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. Prior research on decoding methods, primarily focusing on task-specific models, may not extend to the current era of general-purpose large language models (LLMs). Moreover, the recent influx of decoding strategies has further complicated this landscape. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of LLMs, evaluating their performance, robustness to hyperparameter changes, and decoding speeds across a wide range of tasks, models, and deployment environments. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization. Intriguingly, sensitivity analysis exposes that certain methods achieve superior performance at the cost of extensive hyperparameter tuning, highlighting the trade-off between attaining optimal results and the practicality of implementation in varying contexts.
Forward citations
Cited by 14 Pith papers
-
A First-Principles Theory of Slow Thinking and Active Perception
Active lifting of data distributions via latent-sequence sampling and max-rate uncertainty reduction formally derives slow-thinking LLMs and places them on representation and sampler hierarchies that can be climbed.
-
Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures
Xent Games formalize a large family of LLM evaluation tasks as games whose rewards and constraints are signed cross-entropy sums, and propose using them to build general capability measures.
-
Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities
Most tested LLMs fail to reproduce human implicit causality biases in coreference, coherence, and referring-expression form, even when they show partial coreference effects.
-
Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations
vLLM attention kernel and prefix caching drive model- and task-dependent energy and latency effects, with no universal best config, and can unexpectedly shift measured accuracy.
-
HARP: Hesitation-Aware Reframing in Transformer Inference Pass
HARP improves LLM accuracy by up to 5.16% at about 1.25x inference cost, by running a dropout-perturbed second forward pass only when token-level entropy is high.
-
Adaptive Decoding via Latent Preference Optimization
Adaptive Decoding learns to pick a discrete sampling temperature at token or sequence level via a DPO-style loss over latent temperature choices, and beats fixed temperatures on average across three task families.
-
Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
The paper formalizes test-time scaling into three regimes, introduces a discovery-stability profile for repeated-sampling evaluation, and releases nearly two million reasoning traces.
-
Rare Event Analysis of Large Language Models
Using annealed transition path sampling plus MBAR reweighting, the authors estimate TinyStories-8M completion probabilities for extreme ARI and log-probability values that are unobservable by direct sampling.
-
A Comparative Study of Decoding Strategies in Medical Text Generation
Across five medical text tasks, deterministic decoding methods generally score higher than stochastic sampling, while medical-specific models do not outperform general models and are more sensitive to decoding choice.
-
On Next-Token Prediction in LLMs: How End Goals Determine the Consistency of Decoding Algorithms
The paper claims that no polynomial-time decoder is optimal for all distributions under N-gram Hamming loss, that random sampling is consistent for sequence cross-entropy, and that temperature scaling only works at te...
-
Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding
Fine-tuned Phi-3 language models matched or beat GPT-4o on surgical billing code generation while running locally on four GPUs.
-
Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training
Reinforcement learning post-training gives on-device LLMs an internal 'call for help' action, eliminating external routers and achieving strong math accuracy under cloud-use budgets.
-
Context-Enhanced Contrastive Search for Improved LLM Text Generation
CECS extends Contrastive Search with adaptive temperature, candidate thresholds, and penalties, reporting better coherence and diversity, but the description is underspecified.
-
Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
A survey that categorizes speculative decoding methods into draft-centric and model-centric families and discusses deployment challenges.
Discussion (0). Continue with ORCID to comment.