REVIEW 19 cited by
Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Inference-time computation is a powerful paradigm to enhance the performance of large language models (LLMs), with Best-of-N sampling being a widely used technique. However, this method is computationally expensive, requiring both (1) an external reward model and (2) the generation of multiple samples. In this work, we introduce a new generative self-evaluation scheme designed to adaptively reduce the number of generated samples while maintaining or even improving performance. We use a generative reward model formulation, allowing the LLM to predict mid-generation the probability that restarting the generation will yield a better response. These predictions are obtained without an external reward model and can be used to decide whether or not to generate more samples, prune unpromising samples early on, or to pick the best sample. This capability is very inexpensive as it involves generating a single predefined token. Trained using a dataset constructed with real unfiltered LMSYS user prompts, Llama 3.1 8B's win rate against GPT-4 on AlpacaEval increases from 21% to 34% with 16 samples and math performance on GSM8K improves from 84% to 91%. By sampling only when the LLM determines that it is beneficial to do so and adaptively adjusting temperature annealing, we demonstrate that 74% of the improvement from using 16 samples can be achieved with only 1.2 samples on average. We further demonstrate that 50-75% of samples can be pruned early in generation with minimal degradation in performance. Overall, our methods enable more efficient and scalable compute utilization during inference for LLMs.
Forward citations
Cited by 19 Pith papers
-
Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade
Hidden-state probes can flag doomed LLM-agent episodes from the first round, and a recall-calibrated cascade of abort gates cuts generated tokens by up to 60% while preserving a chosen success-recall target.
-
Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
Matched output limits still produce very different execution-failure mixtures across model families, so benchmark accuracy alone hides how models fail.
-
Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains
Cost-stratified thresholding beats stronger global online optimization for budgeted LLM verification, because uncertainty quality is heteroskedastic across cost strata.
-
Energy-Based Transformers are Scalable Learners and Thinkers
Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...
-
Temporal Sampling for Forgotten Reasoning in LLMs
Sampling test-time responses from multiple training checkpoints, rather than only the final checkpoint, recovers temporarily forgotten correct answers and improves Pass@k by 4-19 points on math reasoning benchmarks.
-
Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection
Route-To-Reason learns to predict the accuracy and token usage of each model-strategy pair and routes every query to the pair that maximizes a tunable accuracy-cost score.
-
TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
TrimR uses a lightweight verifier to detect redundant thoughts in reasoning models and truncates them, cutting runtime by up to 70% without retraining.
-
S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models
S-GRPO trains reasoning models to answer from truncated thinking paths with decaying rewards for earlier correct answers, cutting output length by 35-61% while slightly raising accuracy.
-
An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint
Under strict output token budgets, LLM rankings change: smaller instruction-tuned models can beat larger or reasoning-specialized models, and mid-sized models are latency-optimal.
-
An Annotated Reading of 'The Singer of Tales' in the LLM Era
LLM generation resembles oral-formulaic composition: single-pass, pattern-based, and non-authorial, so AI output should be treated as a new post-literate medium.
-
Teaching LLMs to Refine with Tools
CaP trains LLMs to fix chain-of-thought math solutions by producing program-of-thought code, and shows that DPO preference optimization is essential for the refinement to actually improve accuracy.
-
Quantum Circuit Generation via test-time learning with large language models
An LLM with memory, score feedback, and restart-from-best finds high-entanglement quantum circuits, reaching Meyer-Wallach 1.0 on 25 qubits within 45 queries.
-
Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
The survey's L1/L2 taxonomy and benchmark show that current reasoning models waste compute on easy problems and underthink hard ones, motivating more adaptive inference.
-
DynScaling: Efficient Verifier-free Inference Scaling via Dynamic and Integrated Sampling
DynScaling improves verifier-free inference-time scaling by merging parallel and sequential sampling and allocating budget across queries with a UCB-based uncertainty rule.
-
SLOT: Sample-specific Language Model Optimization at Test-time
SLOT adapts an LLM to each prompt by optimizing a lightweight final-layer vector to minimize prompt loss, boosting benchmark reasoning accuracy by a few points.
-
Reasoning Language Models: A Blueprint
A modular blueprint and open-source framework (x1) that presents existing reasoning language model designs as special cases of one unified toolbox.
-
Efficiently Scaling LLM Reasoning with Certaindex
Certaindex measures when an LLM's intermediate answers stop changing, enabling early exit and dynamic token allocation that cuts token usage by up to 50% with no accuracy drop in tested workloads.
-
TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
TinyMusician distills MusicGen and applies hand-picked mixed-precision quantization to make a 1.04 GB on-device music generator, but the headline '93% quality, 55% smaller' claims conflict with the paper's own tables.
-
Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
Small LLMs with compute-optimal test-time scaling can outperform much larger models on math benchmarks, but the reported strategy is selected on the same test sets used for evaluation.
Discussion (0). Continue with ORCID to comment.