arxiv: 2605.07686 · v1 · submitted 2026-05-08 · 💻 cs.LG

Recognition: no theorem link

The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits

Wenhua Nie , Junlin Liu , Jianan Wu , Zijie Meng , Yilong Fan , Zhang Zijian , Haoran Zheng , Jyh-Shing Roger Jang

Authors on Pith no claims yet

Pith reviewed 2026-05-11 03:24 UTC · model grok-4.3

classification 💻 cs.LG

keywords reasoningaccuracybudgetsmath-500reachestracesalphaanswer

0 comments

The pith

Sharing one fixed token budget between chain-of-thought reasoning and the final answer reduces accuracy because long traces crowd out the answer they support.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Chain-of-thought is usually assumed to help by allowing longer thinking, but when traces and answers must fit inside the same fixed output limit, longer traces can push out the answer. On standard math benchmarks like GSM8K and MATH-500, simply skipping the thinking step matches or beats the full chain-of-thought version at every budget size up to 2048 tokens. The paper introduces a simple truncation-waste formula that decomposes thinking accuracy into the chance a full chain fits in the budget times its accuracy plus the chance it gets cut short times the short-chain accuracy. This formula predicts the point where thinking starts to help and explains why larger models sometimes show worse scaling. Split-budget approaches that give separate limits to reasoning and answering raise accuracy substantially, reaching over 83 percent on the full MATH-500 set.

Core claim

What carries the argument

The truncation-waste decomposition Acc_think(b)=α_c F_L(b)+α_t(1-F_L(b)), which expresses thinking-mode accuracy as a mixture of full-chain accuracy weighted by the fraction of chains that fit inside budget b and truncated-chain accuracy weighted by the fraction that do not.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Models could benefit from generating internal reasoning that does not consume the output token budget at all.
The decomposition offers a practical way to decide when to trigger chain-of-thought based on predicted chain length and available budget.
Similar crowding effects may appear whenever multiple required outputs compete for a fixed generation length, such as multi-step plans or tool-use sequences.

Load-bearing premise

Differences in accuracy between thinking and non-thinking modes are driven primarily by whether the reasoning trace is truncated within the shared budget.

What would settle it

Measure whether thinking-mode accuracy exceeds non-thinking accuracy once the total budget is expanded until nearly all observed reasoning traces fit completely without truncation.

Figures

Figures reproduced from arXiv: 2605.07686 by Haoran Zheng, Jianan Wu, Junlin Liu, Jyh-Shing Roger Jang, Wenhua Nie, Yilong Fan, Zhang Zijian, Zijie Meng.

**Figure 1.** Figure 1: The Coupling Tax. Non-thinking mode (blue) dramatically outperforms thinking mode (orange) at every matched token budget ≤512 on GSM8K (Qwen3-8B, n=1,319). At budget 512, nothink@512 achieves 93.1% while think@512 reaches just 56.9%—a +36.2 pp gap. The gap widens for the 27B model (Table ??). The thinking tax. Through experiments on GSM8K, MATH-500, and five BBH tasks with three Qwen-family sizes (8B, 9B, … view at source ↗

**Figure 2.** Figure 2: Four-tier difficulty distribution of GSM8K problems under Qwen3-8B thinking mode. 31.8% of problems are impossible at all tested budgets. H Mrsd Routing Ablation [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗

**Figure 3.** Figure 3: The thinking tax worsens with model size. At b=512, thinking-mode accuracy collapses with model size (8B: 56.9%, 9B: 15.5%, 27B: 18.4%)—the 9B and 27B taxes are both ∼2.1× larger at this budget (vs. uniform nothink: 93.1%, 93.2%, 95.5%). The 8B non-thinking baseline (green dashed) dominates all thinking configurations at every budget ≤512. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗

**Figure 4.** Figure 4: Town baseline inference pipeline. Stage 1 probes with non-thinking mode at budget B1. If the model stops early (88.8% of GSM8K), the answer is accepted at ∼133 tokens (94.4% accuracy). Otherwise, Stage 2 routes to thinking mode at budget B2, recovering additional correct answers on hard problems. Overall: 90.9% accuracy at 199 average tokens on the full test set (n=1,319). Note: Mrsd extends this with deco… view at source ↗

**Figure 5.** Figure 5: Chain-length CDF FL(b) for Qwen3-8B vs. Qwen3.5-9B on GSM8K (think@2048, n=1,319). The 9B CDF shifts right (stochastic dominance): median chain length increases from 540 tokens (8B) to 993 tokens (9B), with natural-stop rates of 92.8% vs. 56.3%. At any fixed budget b, FL9B (b) ≤ FL8B (b), meaning more 9B chains are truncated— consistent with the observed 2.1× tax ratio at b=512 predicted by Proposition 8. … view at source ↗

**Figure 6.** Figure 6: MATH-500 pilot Pareto frontier. Accuracy vs. average tokens for plotted deterministic methods (Qwen3-8B). Think (orange) is dominated at every budget below the crossover. Nothink (blue) saturates early in this run. IRIS (red) illustrates the split-budget tradeoff; stochastic SC baselines are analyzed separately in [PITH_FULL_IMAGE:figures/full_fig_p039_6.png] view at source ↗

read the original abstract

Chain-of-thought reasoning is often treated as a monotone way to improve language-model accuracy by letting a model think longer. We identify a countervailing effect, the coupling tax: when reasoning traces and final answers share one output-token budget, long traces can crowd out the answer they are meant to support. Across GSM8K, MATH-500, and five BIG-Bench Hard tasks with Qwen3 models at three scales, non-thinking mode matches or outperforms thinking mode on GSM8K and MATH-500 at every budget up to 2048 tokens, while harder tasks shift the crossover to larger budgets. We derive a truncation-waste decomposition, $\mathrm{Acc}_{\mathrm{think}}(b)=\alpha_c F_L(b)+\alpha_t(1-F_L(b))$, that predicts this crossover from chain-length and accuracy statistics and explains inverse scaling within the Qwen family. A DeepSeek-R1-Distill-Llama-8B replication shows the same pattern under a different thinking interface. As a mitigation, split-budget generation decouples reasoning and answer budgets; on full MATH-500, IRIS reaches 74.0% accuracy, a strengthened extraction variant reaches 78.8%, and a fixed non-oracle SC+IRIS gate reaches 83.6%. The results show that test-time reasoning should be evaluated as a budget-allocation problem, not only as a question of whether longer traces are available.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Shared output budgets create a real coupling tax that can make visible CoT worse than direct answers on easier tasks, with a simple decomposition that explains the crossover points.

read the letter

The key takeaway is that under a single fixed token budget, long reasoning traces can starve the final answer of tokens, so non-thinking mode matches or beats thinking mode on GSM8K and MATH-500 up to 2048 tokens while harder tasks push the crossover higher. The paper backs this with results across Qwen3 scales and a DeepSeek replication, plus split-budget mitigations that lift MATH-500 accuracy to 74-83.6% depending on the variant.

Referee Report

2 major / 2 minor

Summary. The paper identifies a 'coupling tax' arising when reasoning traces and final answers share a fixed output token budget, causing long traces to crowd out answers. Empirical results across GSM8K, MATH-500, and five BIG-Bench Hard tasks with Qwen3 models at three scales show non-thinking mode matching or outperforming thinking mode on easier tasks up to 2048 tokens, with crossover shifting to larger budgets on harder tasks. A truncation-waste decomposition Acc_think(b)=α_c F_L(b)+α_t(1-F_L(b)) is derived to predict crossovers from chain-length and accuracy statistics and to explain inverse scaling; a DeepSeek-R1 replication confirms the pattern. Split-budget generation is proposed as mitigation, reaching up to 83.6% on MATH-500.

Significance. If the results hold, the work is significant for reframing test-time reasoning as a budget-allocation problem rather than assuming monotonic gains from longer traces. The breadth of evaluation across tasks, model scales, and the replication with a different thinking interface (DeepSeek-R1-Distill-Llama-8B) provides solid empirical grounding. The decomposition supplies a mechanistic account, and the split-budget experiments demonstrate concrete improvements. These elements could shape future evaluations of CoT and inference-time scaling.

major comments (2)

[Truncation-Waste Decomposition] The truncation-waste decomposition Acc_think(b)=α_c F_L(b)+α_t(1-F_L(b)) (abstract and main derivation) treats α_t as a fixed scalar. However, when a trace is truncated at budget b, remaining tokens for the answer equal b minus the generated prefix length. The conditional distribution of chain lengths L | L > b shifts toward longer chains as b increases, so the typical remaining budget (and thus achievable accuracy in the truncated regime) changes with b. Treating α_t as invariant therefore introduces an approximation whose error grows with chain-length variance and answer-accuracy sensitivity to token count. This directly affects the predicted crossover points, which are central to the explanatory claim.
[Parameter Estimation and Prediction] α_c, α_t, and F_L(b) are estimated from the same experimental data used to observe the performance patterns. The paper should demonstrate that the decomposition yields independent predictions (e.g., via held-out budgets, different models, or out-of-sample validation) rather than largely re-expressing the observed statistics.

minor comments (2)

[Experimental Details] Provide explicit details on how F_L(b) is computed from chain-length statistics, how α_c and α_t are fitted, and the precise prompt templates and accuracy measurement protocols for thinking versus non-thinking modes.
[Figures and Results] Add error bars, confidence intervals, or statistical significance tests to the accuracy-vs-budget plots to support claims that non-thinking matches or exceeds thinking mode.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight important assumptions in our truncation-waste decomposition and the need for stronger validation of its predictions. We respond to each major comment below and will incorporate revisions to address the concerns.

read point-by-point responses

Referee: [Truncation-Waste Decomposition] The truncation-waste decomposition Acc_think(b)=α_c F_L(b)+α_t(1-F_L(b)) (abstract and main derivation) treats α_t as a fixed scalar. However, when a trace is truncated at budget b, remaining tokens for the answer equal b minus the generated prefix length. The conditional distribution of chain lengths L | L > b shifts toward longer chains as b increases, so the typical remaining budget (and thus achievable accuracy in the truncated regime) changes with b. Treating α_t as invariant therefore introduces an approximation whose error grows with chain-length variance and answer-accuracy sensitivity to token count. This directly affects the predicted crossover points, which are central to the explanatory claim.

Authors: We agree that α_t is an approximation averaging over varying remaining budgets in the truncated regime. Our empirical observation that non-thinking accuracy saturates quickly with added tokens supports treating it as roughly constant for first-order predictions. In revision we will add: (i) explicit discussion of the approximation and its assumptions, (ii) quantification of remaining-budget variance across b, and (iii) a sensitivity analysis comparing constant-α_t predictions against a b-dependent α_t(b) fitted from non-thinking curves. This will bound the approximation error while retaining the decomposition's explanatory value for the observed crossovers and inverse scaling. revision: yes
Referee: [Parameter Estimation and Prediction] α_c, α_t, and F_L(b) are estimated from the same experimental data used to observe the performance patterns. The paper should demonstrate that the decomposition yields independent predictions (e.g., via held-out budgets, different models, or out-of-sample validation) rather than largely re-expressing the observed statistics.

Authors: We acknowledge that the current presentation fits parameters on the full observed data. To demonstrate independent predictive utility, the revision will add held-out validation: α_c and α_t will be estimated only on budgets ≤1024 tokens and then used to predict Acc_think(b) at 2048 and 4096 tokens. We will also report cross-model validation by fitting on Qwen3 data and predicting the DeepSeek-R1-Distill-Llama-8B crossovers (and vice versa). These results will appear in a new subsection on predictive validation of the decomposition. revision: yes

Circularity Check

1 steps flagged

Truncation-waste decomposition reconstructs observed accuracy from fitted statistics

specific steps

fitted input called prediction [Abstract]
"We derive a truncation-waste decomposition, Acc_think(b)=α_c F_L(b)+α_t(1-F_L(b)), that predicts this crossover from chain-length and accuracy statistics and explains inverse scaling within the Qwen family."

α_c and α_t are accuracy statistics measured from complete and truncated traces; F_L(b) is the empirical distribution of chain lengths. The formula therefore reconstructs the observed Acc_think(b) as a weighted average of these fitted values. The predicted crossover is the b where this reconstructed curve equals the separately measured non-thinking accuracy, making the 'prediction' a direct re-expression of the input statistics.

full rationale

The paper's central explanatory device is the truncation-waste decomposition, which expresses Acc_think(b) directly as a mixture of two accuracy parameters and the empirical chain-length CDF, all measured from the same experimental runs. This allows the crossover point to be located by solving the equation against the non-thinking accuracy curve, but the location is therefore determined by the input statistics themselves rather than by any independent mechanism. The decomposition is useful for interpretation yet does not constitute a first-principles prediction; the skeptic note that α_t cannot be treated as constant further weakens any claim of independent grounding. No self-citation or ansatz smuggling is present, so the circularity is limited to the fitted-input pattern.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 1 invented entities

The central claim rests on empirical accuracy and length statistics collected from the experiments plus the assumption that truncation effects dominate accuracy differences.

free parameters (2)

α_c and α_t = data-derived
Accuracy for complete versus truncated reasoning traces, estimated from data
F_L(b) = data-derived
Empirical fraction of chains whose length exceeds budget b

axioms (1)

domain assumption Thinking-mode accuracy is a weighted average of accuracies on complete and truncated traces according to the length distribution.
Direct basis for the truncation-waste decomposition stated in the abstract.

invented entities (1)

coupling tax no independent evidence
purpose: Label for the performance penalty caused by shared token budgets between reasoning and answer
New conceptual term for the observed crowding-out effect

pith-pipeline@v0.9.0 · 5589 in / 1614 out tokens · 78435 ms · 2026-05-11T03:24:45.033342+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages · 5 internal anchors

[1]

NeurIPS , year=

Chain-of-thought prompting elicits reasoning in large language models , author=. NeurIPS , year=

work page
[2]

ICLR , year=

Self-consistency improves chain of thought reasoning in language models , author=. ICLR , year=

work page
[3]

NeurIPS , year=

Tree of thoughts: Deliberate problem solving with large language models , author=. NeurIPS , year=

work page
[4]

2024 , howpublished=

Learning to Reason with. 2024 , howpublished=

work page 2024
[5]

2024 , howpublished=

work page 2024
[6]

Guo, Daya and Yang, Dejian and Zhang, He and Song, Junxiao and Zhang, Runxin and Xu, Runze and Zhu, Qihao and Ma, Shirong and Wang, Peiyi and Bi, Xiao and others , journal=

work page
[7]

Qwen3 Technical Report

Qwen3 Technical Report , author=. arXiv preprint arXiv:2505.09388 , year=. 2505.09388 , archiveprefix=

work page internal anchor Pith review Pith/arXiv arXiv
[8]

Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , journal=. Scaling

work page
[9]

s1: Simple test-time scaling

s1: Simple test-time scaling , author=. arXiv preprint arXiv:2501.19393 , year=

work page Pith review arXiv
[10]

Let's Verify Step by Step

Let's verify step by step , author=. ICLR , year=. 2305.20050 , archiveprefix=

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Math-shepherd: Verify and reinforce

Wang, Peiyi and Li, Lei and Shao, Zhihong and Xu, Runxin and Dai, Damai and Li, Yifei and Chen, Deli and Wu, Yu and Sui, Zhifang , booktitle=. Math-shepherd: Verify and reinforce

work page
[12]

NeurIPS , year=

Self-Refine: Iterative Refinement with Self-Feedback , author=. NeurIPS , year=

work page
[13]

Adaptive Computation Time for Recurrent Neural Networks

Adaptive computation time for recurrent neural networks , author=. arXiv preprint arXiv:1603.08983 , year=

work page internal anchor Pith review arXiv
[14]

ICLR , year=

Universal transformers , author=. ICLR , year=

work page
[15]

NeurIPS , year=

Confident adaptive language modeling , author=. NeurIPS , year=

work page
[16]

EMNLP , year=

Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding , author=. EMNLP , year=

work page
[17]

ICML , year=

Fast inference from transformers via speculative decoding , author=. ICML , year=

work page
[18]

Accelerating Large Language Model Decoding with Speculative Sampling

Accelerating large language model decoding with speculative sampling , author=. arXiv preprint arXiv:2302.01318 , year=

work page internal anchor Pith review arXiv
[19]

CVPR , year=

Rapid object detection using a boosted cascade of simple features , author=. CVPR , year=

work page
[20]

Zhang, Jiajie and Lin, Nianyi and Hou, Lei and Feng, Ling and Li, Juanzi , journal=

work page
[21]

Li, Zheng and Dong, Qingxiu and Ma, Jingyuan and Zhang, Di and Jia, Kai and Sui, Zhifang , journal=

work page
[22]

Chen, Xingyu and Xu, Jiahao and Liang, Tian and He, Zhiwei and Pang, Jianhui and Yu, Dian and Song, Linfeng and Liu, Qiuzhi and Zhou, Mengfei and Zhang, Zhuosheng and Wang, Rui and Tu, Zhaopeng and Mi, Haitao and Yu, Dong , journal=. Do. 2024 , eprint=

work page 2024
[23]

Training Verifiers to Solve Math Word Problems

Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[24]

Measuring mathematical problem solving with the

Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , booktitle=. Measuring mathematical problem solving with the

work page
[25]

Challenging

Suzgun, Mirac and Scales, Nathan and Sch. Challenging. Findings of the Association for Computational Linguistics: ACL 2023 , pages=

work page 2023
[26]

Chen, Lingjiao and Zaharia, Matei and Zou, James , journal=. Frugal

work page
[27]

Reason- ing models can be effective without thinking.arXiv preprint arXiv:2504.09858, 2025

Reasoning Models Can Be Effective Without Thinking , author=. arXiv preprint arXiv:2504.09858 , year=

work page arXiv
[28]

Scalable chain of thoughts via elastic reasoning

Scalable Chain of Thoughts via Elastic Reasoning , author=. arXiv preprint arXiv:2505.05315 , year=

work page arXiv