A power-calibrated statistical framework gives closed-form links from KGW watermark parameters (γ, δ) to detection power and KL distortion, enabling principled Pareto-optimal selection.
Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
This paper introduces a novel Bayesian learning model to explain the behavior of Large Language Models (LLMs), focusing on their core optimization metric of next token prediction. We develop a theoretical framework based on an ideal generative text model represented by a multinomial transition probability matrix with a prior, and examine how LLMs approximate this matrix. Key contributions include: (i) a continuity theorem relating embeddings to multinomial distributions, (ii) a demonstration that LLM text generation aligns with Bayesian learning principles, (iii) an explanation for the emergence of in-context learning in larger models, (iv) empirical validation using visualizations of next token probabilities from an instrumented Llama model Our findings provide new insights into LLM functioning, offering a statistical foundation for understanding their capabilities and limitations. This framework has implications for LLM design, training, and application, potentially guiding future developments in the field.
years
2026 3representative citing papers
GLU is a single-pass unsupervised uncertainty score for LLMs formed by multiplying global hidden-state geometric entropy with local token entropy, shown to match or beat baselines on three model families and six benchmarks while catching failure modes local signals miss.
Perturbing the prefix before next-token prediction, during both training and inference, improves out-of-distribution language-model generation and yields a conditional extrapolation guarantee.
citing papers explorer
-
Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking
A power-calibrated statistical framework gives closed-form links from KGW watermark parameters (γ, δ) to detection power and KL distortion, enabling principled Pareto-optimal selection.
-
Integrating Local and Global Entropy for Uncertainty Quantification in LLMs
GLU is a single-pass unsupervised uncertainty score for LLMs formed by multiplying global hidden-state geometric entropy with local token entropy, shown to match or beat baselines on three model families and six benchmarks while catching failure modes local signals miss.
-
Perturbation is All You Need for Extrapolating Language Models
Perturbing the prefix before next-token prediction, during both training and inference, improves out-of-distribution language-model generation and yields a conditional extrapolation guarantee.