REVIEW 7 cited by
Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
read the original abstract
Numerous capability and safety techniques of Large Language Models (LLMs), including RLHF, automated red-teaming, prompt engineering, and infilling, can be cast as sampling from an unnormalized target distribution defined by a given reward or potential function over the full sequence. In this work, we leverage the rich toolkit of Sequential Monte Carlo (SMC) for these probabilistic inference problems. In particular, we use learned twist functions to estimate the expected future value of the potential at each timestep, which enables us to focus inference-time computation on promising partial sequences. We propose a novel contrastive method for learning the twist functions, and establish connections with the rich literature of soft reinforcement learning. As a complementary application of our twisted SMC framework, we present methods for evaluating the accuracy of language model inference techniques using novel bidirectional SMC bounds on the log partition function. These bounds can be used to estimate the KL divergence between the inference and target distributions in both directions. We apply our inference evaluation techniques to show that twisted SMC is effective for sampling undesirable outputs from a pretrained model (a useful component of harmlessness training and automated red-teaming), generating reviews with varied sentiment, and performing infilling tasks.
Forward citations
Cited by 7 Pith papers
-
Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
Layer-wise entropy collapse depth is a weak per-candidate signal that, compounded inside MCMC power sampling, yields state-of-the-art training-free LLM reasoning accuracy.
-
Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion
CDM amortizes SMC inference for reward-tilted discrete diffusion by training a parameterized twist function on contrastive samples with closed-form kernels.
-
Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
Sequential Monte Carlo sampling from a reward-augmented sequence distribution improves LLM performance on HumanEval by up to 54.9% and MATH500 by up to 8.8%, outperforming standard sampling and GRPO.
-
Sound Probabilistic Safety Bounds for Large Language Models
Guided expansion of a few generation-tree branches yields provably valid but extremely small lower bounds on LLM harm probability; in several reported runs the baseline Monte Carlo estimate is orders of magnitude larger.
-
Efficient Inference for Coupled Hidden Markov Models in Continuous Time and Discrete Space
Proposes Latent Interacting Particle Systems with an efficient parameterization of twist potentials to enable approximate posterior inference for coupled continuous-time hidden Markov models via twisted sequential Mon...
-
Guidance for twisted particle filter: a continuous-time perspective
The Twisted-Path Particle Filter parameterizes twisting functions via neural networks and optimizes them against a path-measure KL divergence to improve continuous-time particle filtering.
-
Uncertainty-Aware Simulation-Based Inference for Operations Research with Large Language Models
Lookahead resampling with entropy- and power-based rewards steers LLM decoding toward OR formulations whose short simulated continuations are most concentrated, giving reported pass@1 gains not yet separated from adde...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.