Pith. sign in

REVIEW 2 major objections 6 minor 30 references

Later entropy collapse inside a language model is a usable quality signal that, combined with sequence likelihood at test time, lifts training-free reasoning accuracy without RL, rewards, or labeled data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 17:38 UTC pith:JISDFBSQ

load-bearing objection Solid training-free decoding paper: late entropy collapse is a weak but real tiebreak that compounds under MCMC; length controls are the main missing check, not a collapse of the claim. the 2 major comments →

arxiv 2607.09693 v1 pith:JISDFBSQ submitted 2026-06-19 cs.LG cs.AI

Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

Zibin Meng , Peng Xie , Kani Chen This is my paper
classification cs.LG cs.AI
keywords test-time scalingpower samplingentropy collapselogit lenstraining-free reasoningMCMC decodingcollapse depthLLM reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that how deep a transformer keeps its internal uncertainty before locking in predictions is a genuine quality signal for reasoning, not just a curiosity of the forward pass. Stronger reasoners—including RL-posttrained ones—keep entropy elevated until later layers; among samples from a single base model, the correct ones tend to collapse later too. The authors turn that pattern into a scalar collapse depth and fold it into a joint scoring objective with ordinary sequence likelihood, then sample from that objective with Metropolis–Hastings power sampling. The per-candidate signal is weak (near chance on its own), but as a bounded tiebreak among near-equal-likelihood candidates it compounds across the sampling trajectory into the strongest training-free accuracy they measure on most model–benchmark pairs. Gains are largest out of domain and on harder splits—exactly where likelihood alone is weakest—at only single-digit-percent wall-clock overhead, and without any training, reward model, or labeled data.

Core claim

Among candidates from a base language model, correct reasoning traces tend to keep logit-lens entropy elevated until deeper layers before collapsing, and this late-collapse signature can be read as a training-free quality signal. Defining a normalized sequence collapse depth D(x) and sampling from the joint target proportional to p(x) to the alpha times exp(beta D(x)) improves over likelihood-only power sampling in every tested cell and yields state-of-the-art training-free accuracy in eleven of twelve, with the largest gains out of domain and on harder splits.

What carries the argument

Collapse depth D(x): the average, over tokens, of the shallowest probed layer at which logit-lens entropy falls below a threshold, normalized by total depth. It enters the DEGS target π(x) ∝ p(x)^α exp(β D(x)), used as a bounded likelihood-tiebreak inside MCMC power sampling (DEGS-MCMC).

Load-bearing premise

Among samples from one base model, later entropy collapse really does mark better answers often enough that compounding it as a small tiebreak improves accuracy rather than just adding noise.

What would settle it

On the same base model and probe grid, if correct and incorrect candidates show no reliable difference in collapse depth (AUC near 0.5, no accuracy climb across D(x) buckets), or if setting the depth weight beta above zero never beats beta equals zero under matched compute, the method’s claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes Depth-Entropy Guided Sampling (DEGS), a training-free test-time decoder that augments power sampling with a sequence-level collapse depth D(x) derived from logit-lens entropy trajectories. Motivated by the observation that RL-posttrained reasoners collapse entropy later than base models, the authors define a joint target π(x) ∝ p(x)^α exp(β D(x)) and instantiate it in an MCMC framework (DEGS-MCMC) and a weaker Best-of-N-Entropy reranker. Across three open-weight 7B models and four reasoning benchmarks, DEGS-MCMC improves over likelihood-only power sampling in all twelve cells, is the strongest training-free method in eleven, and often matches or exceeds an in-house GRPO reference out of domain, at single-digit-percent wall-clock overhead. Supporting analyses include a weak but significant association between D(x) and correctness (Fig. 2), component ablations (Table 2), layer-subset localization (Table 7), and a bounded-perturbation argument (Prop. 8).

Significance. If the base-internal late-collapse signal is genuine and not an artifact of length or probe design, this is a clear contribution to training-free reasoning: it is the first method to use the depth-wise evolution of internal uncertainty as a sequence-level selection criterion, with a clean β-toggle design that isolates the depth term under a matched harness, decoder, and budget. Strengths include the matched-compute protocol (Table 6), the necessity of both α and β terms (Table 2), localization of the signal to deeper layers (Table 7), modest and correctly scoped theory (Props. 1–8, especially Prop. 8 bounding the depth term as a near-tiebreak), and an in-house GRPO reference trained under matched conditions rather than quoted from prior work. The result would matter for open-weight deployment settings where reward models and RL data are unavailable, and it usefully reframes intermediate representations as carriers of quality information not reducible to output likelihood.

major comments (2)
  1. [§5.1, Def. 2, Fig. 2, Table 2, Table 8] The central interpretive claim—that intermediate depth-entropy dynamics supply a quality signal not reducible to likelihood—rests on D(x) tracking correctness via late collapse timing (Def. 2; §5.1; Fig. 2). Def. 2 averages normalized per-token collapse depths, so D(x) is not a raw length sum, but correct CoT traces are often longer and more deliberative; if length correlates with both correctness and mean d_t, the β term can act partly as a soft length prior. Prop. 8 only bounds the score perturbation and does not separate length from collapse timing. Fig. 2 and Table 2 report no length-matched buckets, partial correlations, or length-regressed controls, and the scoring-span options listed in Table 8 (full / completion / reasoning / answer / numeric) are not reported. Please add (i) corr(D(x), length) and partial association of D(x) with correctness controlling for length on the same ca
  2. [§5.2, Table 1] Table 1 reports point accuracies with no per-cell uncertainty, and the twelve-cell sign test is acknowledged as non-independent (shared models and benchmarks). The median ~4-point gain over likelihood-only power sampling is the load-bearing empirical claim. With ~1.9 points binomial SE on a 500-item split, several reported margins (e.g., DEGS vs Power-SMC on math splits: 0.5–2 points) are within noise of a single split. Please report bootstrap or binomial CIs (or at least SE) for the main DEGS-vs-β=0 and DEGS-vs-Power-SMC deltas, and clarify which gains remain significant under a multiple-comparison-aware or hierarchical analysis. Directional uniformity can stay as corroboration; magnitude claims need uncertainty.
minor comments (6)
  1. [Figure 1] Figure 1(e) reports mean collapse layer d_t in absolute index, while Def. 2 uses normalized D(x)∈(0,1]. The caption notes this, but a dual axis or a companion panel with normalized D would reduce confusion when linking motivation to the operative signal.
  2. [Table 1 caption / §5] DeepSeek-Math-7B Base/Low-temperature rows note frequent failure to emit a parsable boxed answer under the standardized protocol. State the parse-failure rate explicitly so readers can separate formatting failures from reasoning failures when comparing to power-sampling methods that may alter answer format rates.
  3. [Algorithm 1, Table 4] Algorithm 1 line 7 lists q(x|x') and q(x'|x) but the proposal is random-index resampling; a one-line formula for the proposal density ratio (as in Table 4) would make the acceptance step self-contained.
  4. [§2] Related work cites DoLa for layer contrast; a short explicit contrast in the main text (DEGS leaves the token distribution unchanged and aggregates full-depth collapse into a sequence reweighting) would help readers who know DoLa place the contribution faster.
  5. [Throughout / Table 8] Typos/style: “DEPTH-ENTROPYGUIDEDSAMPLING” title spacing; “MOTIVATINGOBSERVATION” and similar concatenated headings; “V ocab” in Table 8; arXiv-style line breaks leave some equations hard to parse (e.g., acceptance ratio Eq. 6).
  6. [Figure 1 / Appendix A] Appendix A Figure 3 is valuable; consider moving one non-math overlay (e.g., GPQA) into the main Figure 1 to show the late-collapse pattern is not math-only in the motivation panels.

Circularity Check

0 steps flagged

No circular derivation: collapse depth is label-independent, theory only bounds the score perturbation, and correctness is an external empirical hypothesis.

full rationale

The paper's chain is observational → definitional → empirical, not a closed derivation. Collapse depth D(x) (Defs. 1–2) is computed solely from logit-lens entropy trajectories of the base model; correctness labels never enter the definition. The joint target π(x)∝p(x)^α exp(β D(x)) is an engineered objective, not a claimed first-principles prediction of accuracy. Propositions 1–8 establish only normalizability, MH convergence, cost accounting, and that the depth term is a bounded tiebreak (Prop. 8: reordering only when α|Δlog p|≤β); Remark 2 explicitly disclaims any theoretical claim that larger D causes correctness. That link is tested externally in §5.1/Fig. 2 (correct vs. incorrect D distributions, AUC≈0.6) and by the β=0 vs β=5 toggle under a fixed decoder (Table 1–2). Hyperparameters are defaults with reported search ranges and ablations, not fits renamed as predictions. Citations for power sampling, logit lens, and entropy phases are external (Karan & Du, Nostalgebraist, Wendler, Tan); there is no self-citation uniqueness theorem or ansatz smuggled from the authors' prior work. Length-bias confounds (if any) are a validity concern, not circularity. The method is self-contained against public benchmarks with an independent signal definition.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central claim rests on a small set of free hyperparameters (α, β, τ, L_sub), standard MH theory, and the domain assumption that logit-lens entropy trajectories carry a usable quality signal. Collapse depth and the DEGS target are invented constructs defined in the paper; their link to correctness is empirical, not derived from first principles. No hidden physical constants or circular normalizations.

free parameters (5)
  • likelihood exponent α = 4
    Default α=4 (via temperature 0.25); search range {2,4,6,8}. Controls how sharply likelihood is preferred; chosen by hand within power-sampling practice.
  • depth weight β = 5
    Default β=5; search range {0,1,2,5,10}. Scales the collapse-depth term; β=0 recovers plain power sampling. Primary free knob for the method's contribution.
  • entropy threshold τ = 0.25 nats
    Default 0.25 nats for defining per-token collapse depth; search range 0.05–0.50. Determines when a layer is considered collapsed.
  • layer subset L_sub = 7 evenly spaced layers
    Default evenly spaced 7 layers (e.g. {4,8,...,28} for 28-layer models). Placement strongly affects accuracy (Table 7); count is nearly free. Hand-chosen probe grid.
  • MCMC budget (N, B, T_MCMC) = N=16, B=16, T_MCMC=10
    Defaults N=16 candidates, B=16 blocks, T_MCMC=10 steps (160 proposals). Matched across MCMC-family methods but still free design choices.
axioms (4)
  • standard math Metropolis–Hastings with the stated acceptance ratio converges to π when the proposal is irreducible and aperiodic on a finite support.
    Theorem 1 / Appendix F.2; standard Neal (1993) MCMC theory applied to the DEGS target.
  • domain assumption The logit lens (unembedding applied to intermediate hidden states, after final RMSNorm) yields meaningful layer-wise token distributions whose entropy tracks internal commitment.
    Invoked throughout §3–4 and Definitions 1–2; paper notes the lens is imperfect (limitations, citing Wendler et al.).
  • domain assumption Among samples from a fixed base model, sequences with later average entropy collapse are more likely correct than those that collapse early.
    Central empirical hypothesis of §3, validated weakly in §5.1 Figure 2; load-bearing for any gain over β=0.
  • domain assumption Sequence-level power distribution p(x)^α is a useful sharpening of the base model for reasoning, distinct from per-step temperature sampling.
    Inherited from Karan & Du (2025); Proposition 2 restates the distinction. DEGS builds on this framework rather than re-deriving it.
invented entities (2)
  • Sequence-level collapse depth D(x) independent evidence
    purpose: Scalar summary of how deep layer-wise logit-lens entropy stays elevated before falling below τ; used as the depth-entropy quality signal.
    Definitions 1–2. New construct for this paper; independent_evidence is partial via the correctness correlation in Figure 2, but the quantity is defined only for this method.
  • DEGS target distribution π(x) ∝ p(x)^α exp(β D(x)) no independent evidence
    purpose: Joint objective combining power likelihood with collapse depth for MH sampling and reranking.
    Definition 3 / equation 4. Invented scoring distribution; no existence claim beyond being a well-defined reweighting of p.

pith-pipeline@v1.1.0-grok45 · 27525 in / 3682 out tokens · 39334 ms · 2026-07-14T17:38:04.994392+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals. Recent work shows that sampling from sharpened base-model distributions at test time recovers much of the RL gain, yet existing methods rely solely on output-layer likelihoods and ignore the transformer's internal forward-pass dynamics. We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal. We observe that stronger reasoners -- including RL-posttrained variants -- exhibit a distinctive "late collapse": logit-lens decoded entropy stays elevated until deeper layers before converging. We define a per-sequence collapse depth $D(\mathbf{x})$ and a joint objective $\pi(\mathbf{x}) \propto p(\mathbf{x})^\alpha \exp(\beta D(\mathbf{x}))$ that combines sequence likelihood with this depth-entropy structure, instantiated inside an MCMC power-sampling framework (DEGS-MCMC). Across three open-weight models and four reasoning benchmarks, this near-chance per-candidate signal compounds over the sampling trajectory into state-of-the-art training-free accuracy, with gains largest out of domain and on the harder splits -- exactly where likelihood alone falls short -- at single-digit-percent wall-clock overhead. DEGS narrowly trails an in-house GRPO reference on the math splits GRPO was trained for, yet surpasses it out of domain on GPQA for all three models, without any training, reward model, or labeled data.

Figures

Figures reproduced from arXiv: 2607.09693 by Kani Chen, Peng Xie, Zibin Meng.

Figure 1
Figure 1. Figure 1: Late entropy collapse in stronger reasoners. Panels (a)–(d) plot layer-wise entropy tra [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Collapse depth D(x) as a predictor of correctness, on two representative settings: Qwen2.5- Math-7B on MATH500 (a–c) and DeepMind500 (d–f). Each row shows the distribution of D(x) for correct vs. incorrect candidates, the empirical accuracy of candidates by D(x) quantile bucket, and the predictive statistics (AUC, Spearman ρ, mean gap ∆D¯, pairwise ranking accuracy). In both settings, later collapse is pos… view at source ↗
Figure 3
Figure 3. Figure 3: Complete entropy-versus-layer visualizations across all twelve (model, benchmark) pairs. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 22 linked inside Pith

  1. [1]

    Power-SMC: Low-latency sequence-level power sampling for training-free LLM reasoning.arXiv preprint arXiv:2602.10273,

    Seyedarmin Azizi, et al. Power-SMC: Low-latency sequence-level power sampling for training-free LLM reasoning.arXiv preprint arXiv:2602.10273,

    arXiv

  2. [2]

    Eliciting latent predictions from transformers with the tuned lens

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112,

    Pith/arXiv arXiv

  3. [3]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V . Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

    Pith/arXiv arXiv

  4. [4]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv

  5. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  6. [6]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Pith/arXiv arXiv

  7. [7]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  8. [8]

    Rewarding the unlikely: Lifting GRPO beyond dis- tribution sharpening

    Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting GRPO beyond dis- tribution sharpening. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 25559–25571,

    2025

  9. [9]

    Scalable power sampling: Unlocking efficient, training-free reasoning for LLMs via distribution sharpening.arXiv preprint arXiv:2601.21590,

    Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, and Haitham Bou Ammar. Scalable power sampling: Unlocking efficient, training-free reasoning for LLMs via distribution sharpening.arXiv preprint arXiv:2601.21590,

    arXiv

  10. [10]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec,...

    Pith/arXiv arXiv

  11. [11]

    Reasoning with sampling: Your base model is smarter than you think

    Aayush Karan and Yilun Du. Reasoning with sampling: Your base model is smarter than you think. arXiv preprint arXiv:2510.14901,

    Pith/arXiv arXiv

  12. [12]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Pith/arXiv arXiv

  13. [13]

    Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K

    Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K. Mansinghka. Sequential Monte Carlo steering of large language models using probabilistic programs.arXiv preprint arXiv:2306.03081,

    Pith/arXiv arXiv

  14. [14]

    Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    11 Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    Pith/arXiv arXiv

  15. [15]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Qwen: An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  16. [16]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  17. [17]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  18. [18]

    Spurious rewards: Rethinking training signals in RLVR

    Rulin Shao, Shuyue Stella Li, Rui Xin, et al. Spurious rewards: Rethinking training signals in RLVR. arXiv preprint arXiv:2506.10947,

    Pith/arXiv arXiv

  19. [19]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Pith/arXiv arXiv

  20. [20]

    Outcome-based exploration for LLM reasoning.arXiv preprint arXiv:2509.06941,

    Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for LLM reasoning.arXiv preprint arXiv:2509.06941,

    Pith/arXiv arXiv

  21. [21]

    Bottom-up policy optimization: Your language model policy secretly contains internal policies.arXiv preprint arXiv:2512.19673,

    Yuqiao Tan, Minzheng Wang, Shizhu He, Huanxuan Liao, Chengfeng Zhao, Qiunan Lu, Tian Liang, Jun Zhao, and Kang Liu. Bottom-up policy optimization: Your language model policy secretly contains internal policies.arXiv preprint arXiv:2512.19673,

    Pith/arXiv arXiv

  22. [22]

    Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    Pith/arXiv arXiv

  23. [23]

    Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122,

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122,

    Pith/arXiv arXiv

  24. [24]

    Learning to reason without external rewards.arXiv preprint arXiv:2505.19590,

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards.arXiv preprint arXiv:2505.19590,

    Pith/arXiv arXiv

  25. [25]

    Probabilistic inference in language models via twisted sequential Monte Carlo.arXiv preprint arXiv:2404.17546,

    Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential Monte Carlo.arXiv preprint arXiv:2404.17546,

    Pith/arXiv arXiv

  26. [26]

    The layer-wise entropy is Hl(t) =− P v pl(v|x ≤t) logp l(v|x ≤t)

    applies U at intermediate layers: pl(v|x ≤t) = softmax(Uh l,t)v. The layer-wise entropy is Hl(t) =− P v pl(v|x ≤t) logp l(v|x ≤t). Prior work shows entropy is high in early layers and drops sharply in later layers (Wendler et al., 2024). Power distributions.The power distribution pα(x)∝p(x) α (α≥1 ) sharpens the base distribution. Karan and Du (2025) show...

    2024

  27. [27]

    GRPO reference.Following Karan and Du (2025), our GRPO references are produced rather than quoted: we posttrain each base model with GRPO (Shao et al.,

    de- liberately uses a higher-entropy regime than the decoding runs— N=16 candidates per problem at T=0.8 , top-p=0.9—so that the candidate pool spans a wide range of D(x) and the correlation is not estimated on a near-degenerate set of near-identical samples. GRPO reference.Following Karan and Du (2025), our GRPO references are produced rather than quoted...

    2025

  28. [28]

    The training setup mirrors theirs—we adopt the GRPO implementation of Shao et al

    and evaluate the resulting policy through the same harness as every other method. The training setup mirrors theirs—we adopt the GRPO implementation of Shao et al. (2025) with its default hyperparameters, optimize against a ground-truth math verifier on the MATH training split, and use a group size of16 rollouts per prompt. For the two Qwen backbones this...

    2025

  29. [29]

    F.1 NORMALIZABILITY OF THEDEGSTARGET Proposition 1(Normalizability).Let the support S={x:p(x)>0} be finite (e.g

    withα≥1,β≥0. F.1 NORMALIZABILITY OF THEDEGSTARGET Proposition 1(Normalizability).Let the support S={x:p(x)>0} be finite (e.g. sequences of bounded length over a finite vocabulary). ThenZ= P x∈S p(x)α exp(βD(x)) satisfies 0< Z <∞ , soπ(x) =p(x) α exp(βD(x))/Zis a well-defined probability distribution. Proof. Each term is nonnegative, and positive for at le...

    2025

  30. [30]

    from a pro- posal g with g(x)>0 on S

    F.5 RERANKING APPROXIMATES THEDEGSTARGET Proposition 4(Self-normalized importance sampling).Let {x(i)}N i=1 be drawn i.i.d. from a pro- posal g with g(x)>0 on S. Define importance weights wi ∝π(x (i))/g(x(i)), normalized to sum to one. Then for any bounded test function h, the self-normalized estimator P i wih(x(i)) converges almost surely to Eπ[h] as N→ ...

    1993