Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

Alec Go; Blake JianHang Chen; Chao Zhao; Liqian Peng; Tian Tan; Wei-Lin Chen; Yu Meng; Ziqian Lin

REVIEW 4 major objections 5 minor 1 cited by

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that deep-thinking ratio—the share of tokens whose prediction distributions keep revising into deeper layers—predicts answer correctness better than output length or confidence, and that selecting high-DTR samples matches s

desk verdict Novel DTR metric with promising Think@n, but the headline correlation claim is statistically fragile and some numbers don't line up internally. read the letter →

arxiv 2602.13517 v2 pith:MG7EWNIS submitted 2026-02-13 cs.CL

Wei-Lin Chen , Liqian Peng , Tian Tan , Chao Zhao , Blake JianHang Chen , Ziqian Lin , Alec Go , Yu Meng This is my paper

classification cs.CL

keywords deep-thinkingratiotokenschain-of-thoughttest-timescalingreasoningeffortself-consistencyearlystoppingLLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to replace token count as a proxy for how hard an LLM thought about a problem. It defines deep-thinking tokens: tokens whose internal next-token prediction does not stabilize until deeper layers of the network. The deep-thinking ratio (DTR), the fraction of such tokens in a generated answer, is shown to correlate positively with accuracy across four math and science benchmarks and eight model variants, with average r = 0.683, beating length- and confidence-based measures. The paper then uses DTR as a selection signal in Think@n, a best-of-n sampler that votes among high-DTR samples and stops low-DTR generations early. If the claim holds, reasoning quality can be measured from internal states without external verifiers, and test-time compute can be spent more efficiently.

What carries the argument

The central object is the deep-thinking token. At each generation step, the model's unembedding matrix projects intermediate-layer hidden states to vocabulary distributions; the paper tracks, layer by layer, the Jensen-Shannon divergence between each intermediate distribution and the final-layer distribution. The first layer where the running minimum divergence falls below g = 0.5 is the settling depth; if that depth lies in the final 15% of layers (depth fraction rho = 0.85), the token counts as deep-thinking. DTR is the fraction of deep-thinking tokens in the sequence. This depth-wise convergence signal is what does the work: it is the mechanism that distinguishes 'thinking hard' from 'gen

What would settle it

Recompute DTR-accuracy correlations on individual samples rather than 5-bin quantile averages across all 32 model-benchmark settings, and cross-validate g and rho separately per model; if per-sample correlations fall to the confidence-baseline level, or optimal thresholds vary strongly by model, the universal claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the depth at which a token's predicted distribution converges to the final layer's distribution is a meaningful measure of reasoning effort. A token is 'deep-thinking' when, measured by Jensen-Shannon divergence against the final-layer prediction, it first settles only after passing the deep fraction (rho = 0.85) of the layers, with settling threshold g = 0.5. The ratio of such tokens in a sequence (DTR) correlates with answer correctness more strongly and more consistently than token length, reverse token length, log probability, perplexity, entropy, or self-certainty. The paper also shows that DTR estimated from the first 50 tokens can rank s

Load-bearing premise

The load-bearing premise is that a single pair of thresholds (g = 0.5, rho = 0.85), chosen on one model and benchmark, transfers unchanged to all eight models and four benchmarks; if the optimal thresholds differ by model, the average correlation r = 0.683 is inflated.

Editorial extensions

If this is right

DTR can rank samples in best-of-n selection and majority voting (Think@n) to match or beat self-consistency with roughly 50% lower inference cost.
DTR computed from a 50-token prefix is enough for ranking and early stopping, so unpromising generations can be cut off before full decoding.
Length-based proxies are misleading: longer outputs are negatively correlated with accuracy on average (r = -0.59) in the tested settings.
Confidence-based signals such as log-probability and self-certainty are less stable than DTR across model-benchmark combinations.
DTR is a training-free, task-agnostic internal signal that requires only hidden states and no external verifiers or annotations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper does not pursue is using the same layer-wise convergence signal to stop generation mid-sequence: once a token has settled early, the model could skip later layers for that token, saving compute without changing outputs.
Because the appendix shows DTR levels shift with reasoning-level prompts even when weights are fixed, a normalized or model-relative DTR (e.g., z-scored within a model) would be needed before DTR can be compared across deployments.
DTR could also be plugged into training objectives as a reward term or filter, targeting 'deep but concise' reasoning rather than long chains; the paper only demonstrates inference-time uses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper introduces a genuinely new and cheap internal signal — the fraction of tokens whose predicted distribution only settles in deep layers — and shows it tracks accuracy across a wide model/benchmark grid. The Think@n selection protocol looks practically useful. But the central "outperforms baselines" claim is not statistically grounded, and there are internal inconsistencies that need fixing before I'd trust the numbers.

What is new: DTR is not length or confidence; it uses logit-lens JSD to the final layer to define deep-thinking tokens. That is a clean operationalization, and the empirical sweep (8 models x 4 benchmarks) is broad. The fact that DTR correlates positively with accuracy while token count correlates negatively is a nice demonstration that length is a poor proxy. The Think@n idea — rank candidates by DTR from a 50-token prefix, vote on the top half — matches or beats self-consistency on the two models tested while cutting cost roughly in half. That is a real result if it holds up.

Soft spots. First, the correlation analysis is bin-level: 5 quantile bins, Pearson r on bin means, effectively 3 degrees of freedom, no confidence intervals, no per-sample correlation. The difference between DTR's 0.683 and Self-Certainty's 0.605 could easily be noise. Second, the hyperparameters g=0.5 and rho=0.85 were chosen by sweeping on GPQA-D with OSS-20B-high, and that exact cell is the highest in Table 1. That inflates the average. Third, there are outright inconsistencies: Figure 1's DTR values for OSS-120B-medium (avg 0.828, values like 0.715 and 0.795) do not match Table 1's row (0.533 and 0.823, avg 0.786). Section 3.2 reports r=0.962 for the chosen settings on OSS-20B-high GPQA-D, while Table 1 says 0.992 for that cell. The Think@n validation uses Qwen3-4B, which is not in the main correlation table at all, plus one GPT-OSS config. The paper also says DTR "remains positive across models and benchmarks" but Table 1 has two negative cells. These are fixable, but they are the difference between a convincing paper and a suggestive one.

Bottom line: the idea is worth engaging with. This is a paper for people working on test-time scaling and efficient reasoning; it should go to peer review, not be desk-rejected. The authors should add bootstrap CIs and per-sample correlations, validate Think@n on the full model set, and align figures with tables. If those come back clean, it is a solid contribution. For now, I would treat the central comparative claim as not proven.

Referee Report

4 major / 5 minor

Summary. The paper proposes deep-thinking ratio (DTR), a per-token measure of internal inference effort derived from the layer at which an intermediate next-token distribution converges to the final-layer distribution. Across eight model configurations and four reasoning benchmarks, it reports that DTR correlates more strongly with answer accuracy than token count, reverse token count, and several confidence-based baselines, with an average Pearson correlation of r=0.683. It then introduces Think@n, a test-time scaling strategy that selects and aggregates the highest-DTR samples, claiming it matches or exceeds self-consistency while cutting inference cost by roughly half via early stopping on short prefixes.

Significance. If the central correlation claim were statistically established, DTR would be a valuable addition to the toolkit for characterizing reasoning effort, with potential practical benefits for test-time scaling. The paper is well-motivated, the operationalization is mechanistically plausible, and the evaluation is broad in its coverage of models and benchmarks. The appendices provide useful ablations on distance metrics, hyperparameters, and reasoning levels, and the qualitative examples are informative. However, the headline correlation is computed on five quantile-bin means per cell, giving only three degrees of freedom, and no uncertainty estimates or significance tests are reported. The central comparative claim therefore currently rests on fragile statistical ground. The Think@n validation is also limited to two model configurations, one of which is absent from the main correlation analysis.

major comments (4)

[§3.1, Table 1] The central claim that DTR 'substantially outperforms' baselines (average r=0.683 vs. Self-Certainty 0.605) is computed as Pearson r over 5 quantile-bin means per model-benchmark cell (df=3). With only five points per cell, the sampling noise of each r is large, and binning compresses within-bin variance, typically inflating |r|. The paper reports no confidence intervals, no per-sample correlations, and no paired significance test across the 32 cells. The 0.078 margin over Self-Certainty may be entirely due to noise. Please report per-response correlations (e.g., point-biserial or rank correlation between DTR and correctness), bootstrap confidence intervals for each cell and for the average, and a paired test over model-benchmark cells comparing DTR with the best baseline.
[§3.2, Figure 4] The hyperparameters (g=0.5, ρ=0.85) defining DTR were selected by sweeping on GPQA-D with GPT-OSS-20B-high (Figure 4). Table 1 shows that exact cell reports r=0.992, the highest correlation in the entire table. Thus at least one of the 32 cells is optimistically biased by selection on the test set, and the average r=0.683 is inflated. The authors should either report the full sensitivity of Table 1 to the hyperparameter choice, use a validation split for threshold selection, or show that the superiority of DTR over baselines holds across a plausible grid of (g, ρ) values.
[§4, Table 2] Think@n is validated on only two model configurations, OSS-120B-medium and Qwen3-4B-Thinking. However, Qwen3-4B-Thinking does not appear in the main correlation table (Table 1), which instead uses Qwen3-30B-Thinking. No correlation evidence is provided to justify applying DTR to Qwen3-4B. In addition, the prefix-length ablation in Table 3 is reported only for AIME 25 (apparently OSS-120B-medium); the choice ℓprefix=50 is not tested on the other benchmark or on Qwen3-4B. The claim that Think@n 'matches or exceeds' self-consistency across benchmarks would be more convincing if the correlation between DTR and accuracy were shown for the exact model used in the scaling experiments.
[Appendix B] Appendix B states that DTR 'might not be directly comparable across different models or model modes,' and Figure 7 shows that higher reasoning levels of GPT-OSS-120B produce lower DTR values while achieving higher accuracy. This is a substantive limitation that qualifies the abstract's and §3.1's wording: DTR is 'consistently positive' only as a within-model, within-mode relative measure, not as an absolute or cross-model measure of thinking effort. The appendix's caveat should be stated prominently in Section 3.1, and the claims should be scoped accordingly.

minor comments (5)

[§3.2] The text calls a higher settling threshold g a 'stricter criterion,' but the opposite is true: a higher g makes it easier for a token to be considered settled early, reducing DTR. The sentence 'a higher g requires the intermediate states to be distributionally far to the final output until reaching deeper layers in the late regime to be considered settle' is garbled and should be rewritten.
[§2.2, Eq. (4), Algorithm 1] If no layer satisfies D_{t,l} ≤ g, then c_t is undefined. Algorithm 1 and Eq. (4) should specify how such never-settling tokens are treated (e.g., classify them as deep-thinking or exclude them).
[§3.1, Table 1] The text says DTR 'remains positive across models and benchmarks,' but Table 1 contains two negative DTR cells (OSS-20B-medium on AIME 2024: -0.192; Qwen3-30B-Thinking on AIME 2024: -0.657). 'Positive in 30 of 32 cells' is accurate; 'remains positive' is not.
[§4] The cost formula for Think@n and Self-Certainty@n is described as 'the total token count of the top η×n ranked samples, plus a fixed prefix overhead of ℓprefix×η×n, which reflects the cost of generating all candidates prior to early termination.' This is conceptually confusing: generating prefixes for all n candidates costs ℓprefix×n, not ℓprefix×η×n. For η=0.5 the formula happens to coincide with the correct cost if the selected samples' full token counts already include their prefixes, but the derivation should be stated unambiguously.
[Figure 4] The three panels in (a) use different DTR x-axis ranges, which makes the comparison of slopes across thresholds difficult. Consider using a common x-axis range or normalizing the DTR values.

Circularity Check

1 steps flagged · score 5.0 of 10

DTR's headline correlation includes a benchmark/model cell whose thresholds were selected by sweeping on that same cell.

fitted input called prediction [Section 3 (Decoding settings), Section 3.2 (Figure 4), Table 1 (GPQA-D / GPT-OSS-20B-high)]
"We uniformly set the settling threshold g=0.5 and the depth fraction ρ=0.85 to define the deep-thinking regime. ... We report results on GPQA-D using GPT-OSS-20B with reasoning level high. ... Among the tested configurations, (g,ρ)=(0.5,0.85) strikes an ideal balance, yielding a reliable trend with high correlation values."

The (g,ρ) pair was selected by sweeping on GPQA-D with GPT-OSS-20B-high and examining the resulting accuracy correlation (Section 3.2). Table 1 then reports DTR's correlation for that exact cell (r=0.992, the highest DTR entry) as part of the headline average r=0.683. That cell is therefore the fitted objective, not an independent confirmation; the claimed margin over Self-Certainty (0.683 vs 0.605) is partly an artifact of tuning on the benchmark family being evaluated. The 31 other cells and Think@n provide independent evidence, and removing the single tuned cell would leave the average near 0.67, so the circularity is partial.

full rationale

The DTR definition itself is not circular: it is a layer-wise JSD-based statistic and is never defined in terms of accuracy, and no equation reduces DTR to correctness. Self-citations are not load-bearing; the key methodological citations (logit lens, early exiting, JSD, binned correlation) are external. The only circularity-adjacent step is the selection of (g=0.5, ρ=0.85) by sweeping on GPQA-D / GPT-OSS-20B-high and then including that same model-benchmark cell in the Table 1 average used for the central claim. This is a fitted-input-called-prediction issue rather than an equation-level identity: neighboring thresholds and the other 31 cells still show positive DTR correlations, so the finding has substantial independent content. Statistical concerns (5-bin Pearson r, no error bars, small df, test-set leakage) are correctness/validity issues and are not scored as circularity beyond the tuning step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

DTR is a new operationalized metric. Its definition depends on two hyperparameters (g, rho) that are fitted to one benchmark. The axioms are standard assumptions from interpretability/early-exit research plus an assumption that binned correlations are representative. No new physical entities are introduced.

free parameters (6)

settling threshold g = 0.5
Chosen by sweeping g in {0.25,0.5,0.75} on GPQA-D with GPT-OSS-20B-high; g=0.5 showed most robust positive correlation (Figure 4a).
depth fraction rho = 0.85
Chosen from {0.8,0.85,0.9,0.95} on same setting; all yield positive slopes, 0.85 selected as balance (Figure 4b).
prefix length for Think@n = 50 tokens
Selected from ablation on AIME25 OSS-120B-medium; 50 achieved highest accuracy with lower cost (Table 3).
top eta for Think@n = 50%
Chosen from {25,50,75}% on AIME25 experiments; 50% peaks (Figure 8b).
number of samples n = 48
Chosen for Think@n experiments; not swept for main claim.
number of quantile bins = 5
Used for binned Pearson correlations following Gema et al.; binning can inflate correlation.

assumptions (5)

domain assumption Applying the unembedding matrix to intermediate-layer hidden states yields meaningful next-token distributions without auxiliary training.
Relies on logit lens / early-exit literature (Nostalgebraist 2020; Belrose 2023), cited in Section 2.1.
domain assumption Jensen-Shannon divergence between intermediate and final-layer distributions is an appropriate measure of distributional settling.
Motivated by Chuang et al. 2023; alternative metrics (KL, cosine) give weaker/less stable correlations (Section A).
domain assumption The final-layer distribution is the converged reference for each token.
By construction D_{t,L}=0; the measure scores how early layers differ from the model's actual sampling distribution.
domain assumption Sampling with temperature=1.0/topp=1.0 and fixed neutral prompts gives natural variation without budget confounds.
Decoding settings in Section 3; used to attribute accuracy differences to thinking effort rather than prompt-induced length.
ad hoc to paper Pearson correlation over 5 quantile bins represents the per-sample DTR-accuracy relationship.
Bin-level correlation averages per-sample outcomes; the paper validates per-sample utility only indirectly via Think@n on 2 models.

invented entities (1)

deep-thinking token
purpose: A token whose predicted distribution does not settle until late layers; used to compute DTR and rank samples.
Operational definition introduced in Section 2.2; evidence is internal correlations and Think@n gains, not an external benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens." pith.science (2026). https://pith.science/paper/MG7EWNIS

@misc{pith2026260213517,
  author       = {Pith},
  title        = {Pith review of: Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MG7EWNIS}},
  note         = {Machine review of arXiv:2602.13517}
}

read the original abstract

Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning
cs.AI 2026-05 conditional novelty 6.0 of 10

Training LLMs to write reusable intermediate states before the context window fills, and optimizing directly for success after reset, improves long-horizon reasoning accuracy under fixed context windows.

Reference graph

Works this paper leans on

3 extracted references · cited by 1 Pith paper

[2]

*Triangle𝐶𝐻𝐺:* Side𝐻𝐺 is vertical with lengthℎ; the horizontal distance from𝐶(− 15, 0) to the line𝑥= 9− 𝑤 2 (the line containing𝐻𝐺) is24− 𝑤

Hence [𝐷𝐺𝐹]= 1 2𝑤 12− ℎ 2 . *Triangle𝐶𝐻𝐺:* Side𝐻𝐺 is vertical with lengthℎ; the horizontal distance from𝐶(− 15, 0) to the line𝑥= 9− 𝑤 2 (the line containing𝐻𝐺) is24− 𝑤
[3]

The problem states[𝐷𝐺𝐹]=[𝐶𝐻𝐺]

Hence [𝐶𝐻𝐺]= 1 2ℎ 24− 𝑤 2 . The problem states[𝐷𝐺𝐹]=[𝐶𝐻𝐺]. Cancelling the factor 1 2 gives 𝑤 12− ℎ 2 =ℎ 24− 𝑤 2 . The terms− 𝑤ℎ 2 appear on both sides and cancel, leaving 12𝑤=24ℎ=⇒𝑤=2ℎ.(2) — Solving for𝑤andℎ Substituting (2) into (1): (2ℎ) 2+ℎ 2 =144=⇒5ℎ 2 =144=⇒ℎ 2 = 144 5 . Hence ℎ= 12√ 5 = 12 √ 5 5 , 𝑤=2ℎ= 24√ 5 = 24 √ 5 5 . The area of rectangle𝐸𝐹𝐺𝐻is...
[2025]

C is closer to GH than to EF

in Table 4 and for GPQA in Table 5. 19 Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens Table 4|Inference prompt for math tasks (AIME 2024, AIME 2025, HMMT 2025). Prompt for AIME 2024, AIME 2025, HMMT 2025 Please reason step by step, and put your final answer within\boxed{}. Table 5|Inference prompt for GPQA. Prompt for G...

2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [2]

*Triangle𝐶𝐻𝐺:* Side𝐻𝐺 is vertical with lengthℎ; the horizontal distance from𝐶(− 15, 0) to the line𝑥= 9− 𝑤 2 (the line containing𝐻𝐺) is24− 𝑤

Hence [𝐷𝐺𝐹]= 1 2𝑤 12− ℎ 2 . *Triangle𝐶𝐻𝐺:* Side𝐻𝐺 is vertical with lengthℎ; the horizontal distance from𝐶(− 15, 0) to the line𝑥= 9− 𝑤 2 (the line containing𝐻𝐺) is24− 𝑤

[2] [3]

The problem states[𝐷𝐺𝐹]=[𝐶𝐻𝐺]

Hence [𝐶𝐻𝐺]= 1 2ℎ 24− 𝑤 2 . The problem states[𝐷𝐺𝐹]=[𝐶𝐻𝐺]. Cancelling the factor 1 2 gives 𝑤 12− ℎ 2 =ℎ 24− 𝑤 2 . The terms− 𝑤ℎ 2 appear on both sides and cancel, leaving 12𝑤=24ℎ=⇒𝑤=2ℎ.(2) — Solving for𝑤andℎ Substituting (2) into (1): (2ℎ) 2+ℎ 2 =144=⇒5ℎ 2 =144=⇒ℎ 2 = 144 5 . Hence ℎ= 12√ 5 = 12 √ 5 5 , 𝑤=2ℎ= 24√ 5 = 24 √ 5 5 . The area of rectangle𝐸𝐹𝐺𝐻is...

[3] [2025]

C is closer to GH than to EF

in Table 4 and for GPQA in Table 5. 19 Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens Table 4|Inference prompt for math tasks (AIME 2024, AIME 2025, HMMT 2025). Prompt for AIME 2024, AIME 2025, HMMT 2025 Please reason step by step, and put your final answer within\boxed{}. Table 5|Inference prompt for GPQA. Prompt for G...

2024