REVIEW 6 cited by
A Simple Model of Inference Scaling Laws
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we propose a simple statistical ansatz based on memorization to study scaling laws in the context of inference, specifically how performance improves with multiple inference attempts. We explore the coverage, or pass@k metric, which measures the chance of success over repeated attempts and provide a motivation for the observed functional form of the inference scaling behavior of the coverage in large language models (LLMs) on reasoning tasks. We then define an "inference loss", which exhibits a power law decay as the number of trials increases, and connect this result with prompting costs. We further test our construction by conducting experiments on a simple generative model, and find that our predictions are in agreement with the empirical coverage curves in a controlled setting. Our simple framework sets the ground for incorporating inference scaling with other known scaling laws.
Forward citations
Cited by 6 Pith papers
-
Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness
A systematic benchmark shows multi-agent debate's value depends on task difficulty, model scale, and agent diversity: limited for math unless problems are hard or models weak, but useful for safety when agents are diverse.
-
CyberV: Cybernetics for Test-time Scaling in Video Understanding
A training-free test-time feedback loop, using attention drift to select key frames, improves video MLLM accuracy, with the largest gains on knowledge-heavy VideoMMMU.
-
X-Factor: Quality Is a Dataset-Intrinsic Property
Across 2,500 class-balanced MNIST subsets and 10 model architectures, test-error Z-scores correlate strongly across models (mean R2=0.82 excluding GNB), supporting dataset quality as an intrinsic property.
-
First Finish Search: Efficient Test-Time Scaling in Large Language Models
First Finish Search launches n parallel reasoning traces and returns the shortest one, improving accuracy on AIME benchmarks while cutting token use.
Discussion (0). Sign in to comment.