Pith. sign in

REVIEW 5 major objections 6 minor 28 references

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ReCo argues that a per-step process reward is the single signal that should coordinate KV-cache compression, reflection-token penalization, and early stopping, cutting generated tokens by 37–65% and latency by 2.08–2.35x while largely…

desk verdict ReCo is a sensible reward-coordinated efficiency system with a useful length-inflation observation, but the test-set-tuned hyperparameters and missing error bars keep the headline gains from being fully out-of-sample. read the letter →

arxiv 2608.04771 v1 pith:5A4JUZXH submitted 2026-08-05 cs.AI

Qiyuan Zhu , Dezhi Li , Pengyu Cheng , Tianle Chen , Jiacheng Wang , Ruijie Shen , Hao Gu , Sida Lin
show 3 more authors
Zirui Liu Jiacheng Liu Sirui Han
This is my paper
classification cs.AI
keywords KVcachecompressionprocessrewardefficientreasoningchain-of-thoughtearlystoppingreflectiontokenslatencyreductionlargemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReCo is a step-wise inference framework that tries to make long chain-of-thought reasoning cheaper by treating the per-step process reward as one control signal for three decisions: how much of the accumulated KV cache to keep, how strongly to penalize redundant reflection tokens, and when to stop reasoning. The central claim is that coordinating these three levers under one reward, rather than compressing the cache alone or shortening the output alone, cuts generated tokens by 37–65% and end-to-end latency by 2.08–2.35x across three reasoning models and six benchmarks, with only small accuracy loss. If true, the practical upshot is that reasoning-model inference can be made substantially cheaper without retraining the model, and that KV-cache compression and generation length cannot be optimized independently on reasoning workloads.

What carries the argument

The central object is the per-step process reward $v_i$ produced by a 30M-parameter estimator distilled from a 7B teacher reward model. It plugs into three mechanisms: (1) a reward-modulated retention ratio $\lambda_i = \lambda - \delta (v_i - \bar{v})/(v_{\max} - v_{\min})$ that sets the budget for attention-guided token eviction from the whole accumulated cache; (2) a reward-banded logit penalty on reflection tokens, active only in the mid and high reward bands; and (3) an early-stopping probe that commits when answer perplexity $\mathrm{PPL}(a) \le \tau_p$ after two consecutive top-band rewards. The identity carrying the argument is the empirical correlation between high step reward and low information loss under token deletion, which licenses using reward as the cache-retention controller.

What would settle it

Take a reasoning trajectory and, at a step where the latest reward is high, evict tokens from the earliest low-reward steps instead of preserving them; if accuracy drops substantially relative to evicting the same budget from the latest high-reward steps, the whole-cache retention rule of Eq. (1) fails. A simpler version is to shuffle the reward signal across steps while keeping all other ReCo components fixed; if the token and latency savings persist while accuracy drops, the reward is not doing the load-bearing work.

Watch

Extended reading notes

Core claim

The paper claims that a reasoning state's tolerance to context loss varies along the trajectory and is tracked by process reward, and that compressing the KV cache on its own systematically lengthens subsequent generation, so the two effects must be managed jointly. ReCo operationalizes this with a lightweight 30M reward estimator that scores each completed reasoning step; that score sets the cache retention ratio (higher reward shrinks the cache harder), selects which tokens survive by attention from a recent window, applies a reward-banded logit penalty to reflection tokens, and triggers early stopping after two consecutive high-reward steps when an answer probe's perplexity is low. The paper reports that this joint scheme stays within roughly 2–3 accuracy points of full-cache CoT while cutting tokens by 37–65% and latency by 2.08–2.35x across three reasoning models and six benchmarks, and that cache-only baselines both lose more accuracy and generate more tokens.

Load-bearing premise

The load-bearing premise is that the reward of the latest completed step describes how safely the entire accumulated KV cache, including earlier low-reward steps, can be shrunk, even though the supporting experiment only tested deleting tokens inside high-reward steps rather than evicting whole-cache tokens from earlier steps, and used the teacher reward model rather than the distilled 30M estimator.

Editorial extensions

If this is right

  • On reasoning workloads, KV-cache compression should be evaluated by end-to-end latency and generated tokens, since cache-only methods can report per-token savings while actually outputting more tokens than the full-cache baseline.
  • A single process-reward signal is enough to coordinate cache retention, generation penalty, and stopping, so step-wise compression need not be uniform across the trajectory.
  • The full ReCo configuration ($\lambda=0.25$, $\delta=0.10$, $\tau_p=1.10$) transfers across three models and six datasets without per-task tuning, suggesting the mechanism is not benchmark-specific.
  • Removing any one of the three components degrades the accuracy-cost trade-off, so the coordinated design, not any single component, carries the reported balance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reward-tolerance correlation generalizes, other cache-reduction mechanisms such as quantization, merging, or low-rank projection could be made step-adaptive in the same way, varying aggressiveness by per-step reward instead of a fixed schedule.
  • The length-inflation measurement, where up to 79.8% of problems generate longer outputs under R-KV compression, implies that published speedups of cache-only methods on reasoning models may be overstated unless they report end-to-end latency.
  • A testable extension is to replace the curated reflection-token set with per-model learned penalty targets, or to make the band thresholds $\tau_\ell$ and $\tau_h$ adaptive per trajectory.
  • Reward and perplexity are complementary in the stopping rule; swapping the perplexity probe for a calibrated confidence model could test whether the same accuracy-length balance persists outside math and science benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes ReCo, a step-wise inference framework for large reasoning models that uses a lightweight process-reward estimator (Pilot) to coordinate three mechanisms: reward-adaptive KV-cache compression (Eq. (1)), a reward-banded logit penalty on reflection tokens (Eqs. (3)-(4)), and confidence-based early stopping (Eqs. (5)-(6)). The authors report 37%-65% token reductions and 2.08x-2.35x end-to-end latency speedups over Full CoT across three models and six benchmarks, with small accuracy losses. The paper also presents two motivation experiments: one showing reward-guided deletion preserves accuracy better than random deletion, and one showing cache compression alone increases generation length.

Significance. Conceptually, the paper makes a useful and falsifiable claim: process reward can serve as a unified control signal for both cache retention and generation length, and compressing the cache without controlling generation can backfire. The motivation experiments are clean, and the component ablation (Fig. 4) provides initial evidence that the three components are complementary. The overhead accounting (Sec. 4.2 footnote) and peak-memory measurement (Table 3) are valuable. However, the central quantitative claim is currently supported mainly by test-set-selected hyperparameters and single-point averages over three runs without variance; until an out-of-sample evaluation and uncertainty quantification are provided, the magnitude of the reported gains should be treated as unverified. No code or machine-checked artifacts accompany the submission.

major comments (5)
  1. [§4.3, Fig. 5, Tables 1-2] The headline results are not out-of-sample. The sensitivity study in §4.3 sweeps λ, δ, and τ_p on AIME25, AMC, and MATH500 with Llama-8B and explicitly selects the configuration that maximizes accuracy on those exact sets (e.g., δ=0.10, τ_p=1.10). Those same three benchmarks then appear in Tables 1 and 2, where the chosen configuration is used to compute the reported 37%-65% token reduction and 2.08x-2.35x speedup. This is test-set selection, not independent evaluation; it can inflate the reported gains. The authors should tune on a disjoint development set (or use nested cross-validation) and report the main results for the selected configuration on held-out benchmarks, or show that the conclusions are stable across the full sweep.
  2. [§3.2, §3.3, §4.1] Several implementation values that are load-bearing for reproducibility are missing. §4.1 specifies λ, δ, w, τ_l, τ_h, and τ_p, but not the reflection penalty strength β of Eq. (3), the cache growth interval S between compressions in §3.2, or the curated reflection-token set R of §3.3. Without these, the reflection-control and compression-scheduling components cannot be reimplemented, and the claimed overhead percentages cannot be checked. The paper should state β and S and provide the full token list (or the exact curation procedure) for R.
  3. [§2.1, §3.1, Eq. (1)] The central mechanism of Eq. (1) is not directly supported by the motivation experiment. The motivation experiment in Sec. 2.1 deletes tokens within high-reward steps, whereas Eq. (1) uses the latest step's reward to set a single retention ratio for the whole accumulated cache, including earlier low-reward steps whose tokens were never tested for eviction. Moreover, the motivation experiment uses the teacher PRM (Skywork-o1-Open-PRM-7B) while the online system uses the distilled 30M Pilot (Sec. 3.1), so the transfer from teacher-based deletion to Pilot-based whole-cache compression is asserted rather than demonstrated. The authors should add an experiment that directly tests whole-cache eviction driven by the latest reward, or modify the method to score retained tokens per step.
  4. [§4.2, Tables 1-2] No uncertainty quantification is reported, despite the claim that 'all reported numbers are averaged over three independent runs' (Table 1 note). With AIME24/25 containing only 30 problems and AMC being similarly small, differences of a few accuracy points between methods (e.g., 33.3% vs. 36.7% on Llama-8B AIME25, a one-problem difference) may be within noise. The paper should report per-run values, standard deviations or confidence intervals, and significance tests (or bootstrap intervals) for the accuracy and latency comparisons, especially for the comparisons to SAT and Dynasor that drive the 'best balance' claim.
  5. [§4.1] The fairness of the latency comparison is not established. The statement that 'SAT and Dynasor are tuned to a comparable level of acceleration' is not accompanied by the tuning procedure, the target speedup, or the resulting hyperparameters. Since the central latency claim is exactly 'ReCo reaches 2.08x-2.35x while preserving accuracy,' the baseline tuning protocol must be specified so the reader can verify the comparison is not conservative for the baselines.
minor comments (6)
  1. [Eq. (1)] In Eq. (1), the denominator (v_max - v_min) can be zero if all step rewards in a trajectory are equal, producing a division by zero; a guard or a stated convention for this case is needed.
  2. [§3.3] The fate of a discarded probe answer is unspecified: if the probe tokens are left in the KV cache, they affect later steps and the reported latency; if they are removed, the removal mechanism should be described.
  3. [§3.1] The phrase 'Pilot retains its step-level discriminability' is not defined; please state a quantitative criterion, such as agreement with the teacher on step rankings.
  4. [Table 3] Table 3 does not state whether peak memory is the mean or the maximum over the three runs, nor the variance across runs.
  5. [§4.2 footnote] The overhead percentages for Pilot scoring, KV compression, reflection penalty, and early stopping are not accompanied by a measurement methodology or error bars, so it is unclear how stable these fractions are.
  6. [§4.1] Please report the random seeds used for the three independent runs to enable exact reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

No equation-level circularity; partial evaluation circularity from test-set hyperparameter selection.

  1. fitted input called prediction [Sec. 4.3 (Hyper-parameter sensitivity) and Tables 1-2]
    "Accuracy peaks at δ = 0.10 (80.0% AMC, 80.6% MATH500, 33.3% AIME25), which we adopt as a robust default. ... τp = 1.10 is best on all three sets (33.3% AIME25, 80.0% AMC, 80.6% MATH500) while already saving many tokens. We use one configuration(λ= 0.25,δ= 0.10,τp = 1.10)forallmodels and datasets, without per-task tuning."

    The hyperparameters λ, δ, and τp are selected by sweeping on AIME25, AMC, and MATH500 and choosing the values that perform best on exactly those datasets. Tables 1 and 2 then present ReCo's accuracy, token, and latency on those same datasets as part of the headline six-benchmark evaluation; the Llama-8B row reports precisely the selected values (AIME25 33.3%, AMC 80.0%, MATH500 80.6%). The reported accuracy-preservation numbers for these benchmarks are therefore the fitted optimum, not an out-of-sample prediction, and the measured token/latency reductions are taken at a configuration chosen on the same test sets.

full rationale

ReCo's derivation chain is not self-definitional. Equation (1) maps an external process-reward estimate to a retention ratio, and the reward comes from Pilot, a fixed 30M estimator, not from the target answers; the reflection penalty and early-stopping probe likewise depend on external signals (reward and answer perplexity). The Sec. 2.1 observation and the method share the same reward family, but that is an empirical design choice, not a logical identity, and the observation is tested rather than assumed. No uniqueness theorem is imported, no ansatz is smuggled in via citation, and the many self-citations (e.g., Chen et al. 2026) are background or motivational, not load-bearing. The single substantive circularity concern is the Sec. 4.3 hyperparameter sweep: λ, δ, and τp are tuned on AIME25, AMC, and MATH500, and the same three sets later appear in Tables 1-2, so the headline numbers for those sets are in-sample. Because GSM8K, AIME24, GPQA, and the Qwen-7B/Qwen3-8B evaluations remain out-of-sample, the central claim is only partially compromised, not forced.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method leans on a pre-trained reward estimator (Pilot) and on several unstated hyperparameters that are either tuned on the test benchmarks or absent from the paper. The central causal premise (reward tracks compression tolerance) is an empirical domain assumption that is only partially tested.

free parameters (8)
  • lambda (base retention ratio) = 0.25
    Selected from a sensitivity sweep on AIME25, AMC, and MATH500 with Llama-8B (Fig. 5a); controls the default cache size.
  • delta (reward adjustment range) = 0.10
    Tuned in Fig. 5b; at delta=0 the compression policy is uniform, and the reported accuracy peaks at delta=0.10.
  • tau_p (early stop perplexity threshold) = 1.10
    Tuned in Fig. 5c; balances token savings against accuracy, with MATH500 dropping from 80.6% to 68.4% when tau_p is raised to 1.20.
  • tau_l, tau_h (reward band thresholds) = 0.4, 0.8
    Chosen without reported justification; they determine when reflection penalties activate and when early stopping is triggered.
  • beta (reflection penalty strength)
    Not reported in the paper; required for implementing Eq. (3) and Eq. (4).
  • S (cache growth interval between compressions)
    Defined in Sec. 3.2 but no value is given; controls how often the cache is compressed.
  • w (attention window) = 32
    Chosen for attention-guided token selection in Eq. (2); no sensitivity analysis reported.
  • Reflection token set R (curated list)
    Not a scalar but a hand-curated set (e.g., 'Wait', 'Hmm') with tokenizer variants, listed nowhere in the paper; essential for the logit penalty.
assumptions (5)
  • domain assumption The reward of the latest step summarizes the compression tolerance of the whole accumulated context, not just the current step.
    Used to justify Eq. (1), but Sec. 2.1 only tested deleting tokens within high-reward steps, not whole-cache eviction, and with the teacher PRM rather than Pilot.
  • domain assumption Reasoning steps defined by newline tokens are coherent units suitable for per-step control.
    Adopted from Huang et al. 2026 without validation in this paper; used to define when rewards are computed.
  • domain assumption Attention scores from the last w tokens identify the most useful cache tokens for future reasoning.
    Standard eviction assumption inherited from SnapKV; not independently validated.
  • domain assumption Perplexity of a tentative answer reflects confidence in the final answer.
    Used in Eq. (5) and (6); the paper argues it complements reward but provides no calibration analysis.
  • domain assumption Suppressing reflection tokens via a logit penalty does not damage reasoning quality when reward is high.
    Motivated by prior work (Wang et al. 2025; Lotfi et al. 2026) and the ablation in Fig. 4, but penalty strength and token list are unreported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning." pith.science (2026). https://pith.science/paper/5A4JUZXH

@misc{pith2026260804771,
  author       = {Pith},
  title        = {Pith review of: Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5A4JUZXH}},
  note         = {Machine review of arXiv:2608.04771}
}
read the original abstract

Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache. Two observations point the other way. First, a reasoning state's tolerance to context loss varies along the trajectory, and process reward tracks it: deleting tokens at high-reward steps preserves accuracy far better than deleting the same budget at random. Second, compression is not free on the generation side, since a smaller cache leads the model to generate more tokens, partly canceling the saving. Together these motivate coordinating both sides under a single process reward. We propose ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: (1) reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, (2) a reward-banded penalty on reflection tokens that curbs redundant generation, and (3) confidence-based early stopping that triggers when the reasoning is reliable. Across three reasoning models and six benchmarks, ReCo reduces generated tokens by 37%-65% and end-to-end latency by 2.08x-2.35x over Full CoT, all while largely preserving accuracy.

Figures

Figures reproduced from arXiv: 2608.04771 by the authors.

Figure 1
Figure 1. Introduction overview on DeepSeek-R1-Distill [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of ReCo. After each step ci , a process reward vi drives three components: ❶ KV Compression sets a reward-modulated retention ratio, shrinking the cache harder at high-reward steps; ❷ Reflection Control maps vi to a reward￾banded logit penalty βi on reflection tokens; and ❸ Early Stopping halts once answer perplexity satisfies PPL(a) ≤ τp. nated by the model’s own growing chain of thought, whose content mus… view at source ↗
Figure 3
Figure 3. Motivating observations for reward-guided KV-cache compression and generation control. (a) Deleting tokens from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Component ablation on DeepSeek-R1-Distill [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Hyper-parameter sensitivity of ReCo (Llama-8B). [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 7 canonical work pages

  1. [3]

    Chang,C.-C.;Lin,W.-C.;Lin,C.-Y.;Chen,C.-Y.;Hu,Y.-F.; Wang,P.-S.;Huang,N.-C.;Ceze,L.;Abdelfattah,M.S.;and Wu,K.-C.2024

    xkv: Cross-layer svd forkv-cachecompression.arXivpreprintarXiv:2503.18893. Chang,C.-C.;Lin,W.-C.;Lin,C.-Y.;Chen,C.-Y.;Hu,Y.-F.; Wang,P.-S.;Huang,N.-C.;Ceze,L.;Abdelfattah,M.S.;and Wu,K.-C.2024. Palu:Compressingkv-cachewithlow-rank projection.arXiv preprint arXiv:2407.21118. Chen,T.;Cheng,P.;Zhu,Q.;Wang,J.;Liu,B.;Gu,H.;Shen, R.; Hou, X.; Han, S.; and Liu, J

    arXiv 2024

  2. [4]

    Chen, Z.; Wei, Z.; Bai, Y.; Xiong, X.; and Wu, J

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187. Chen, Z.; Wei, Z.; Bai, Y.; Xiong, X.; and Wu, J

    arXiv

  3. [5]

    InFindings of the Association for Computational Linguistics: ACL 2025, 21539–21564

    Tagrouter: Learning route to llms through tags for open- domain text generation tasks. InFindings of the Association for Computational Linguistics: ACL 2025, 21539–21564. Cheng,Z.;Chen,D.;Fu,M.;andZhou,T.2026. Optimizing length compression in large reasoning models. InProceed- ingsofthe64thAnnualMeetingoftheAssociationforCom- putationalLinguistics(Volume1...

    work page 2025

  4. [7]

    Feng,S.;Fang,G.;Ma,X.;andWang,X.2025

    Gemini 2.5: Pushing the fron- tier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261. Feng,S.;Fang,G.;Ma,X.;andWang,X.2025. Efficientrea- soning models: A survey.arXiv preprint arXiv:2504.10903. Feng, Y.; Lv, J.; Cao, Y.; Xie, X.; and Zhou, S. K

    arXiv 2025

  5. [9]

    InFindings of the Asso- ciation for Computational Linguistics: ACL 2026, 17051– 17064

    Qarl: Rollout- aligned quantization-aware rl for fast and stable training un- der training–inference mismatch. InFindings of the Asso- ciation for Computational Linguistics: ACL 2026, 17051– 17064. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al

    work page 2026

  6. [10]

    Han,I.;Kacham,P.;Karbasi,A.;Mirrokni,V.;andZandieh, A.2025a

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Han,I.;Kacham,P.;Karbasi,A.;Mirrokni,V.;andZandieh, A.2025a. Polarquant:Quantizingkvcacheswithpolartrans- formation.arXiv preprint arXiv:2502.02617. Han, T.; Wang, Z.; Fang, C.; Zhao, S.; Ma, S.; and Chen, Z. 2025b. Token-budget-aware llm re...

    arXiv 2025

  7. [11]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874. Hooper,C.;Kim,S.;Mohammadzadeh,H.;Mahoney,M.W.; Shao,Y.S.;Keutzer,K.;andGholami,A.2024.Kvquant:To- wards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37: 1270–1303. Hu,X.;Wei,X.;Gu,H.;Zhang,...

    arXiv 2024

  8. [12]

    Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

    Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling. arXiv preprint arXiv:2607.02980. Huang,W.;Bai,X.;Chen,K.;Chen,X.;Chen,Y.;Guan,W.; andZhang,M.2026. Sat:Balancingreasoningaccuracyand efficiency with stepwise adaptive thinking. InProceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: L...

    work page Pith review arXiv 2026

Show all 28 references
  1. [13]

    Li, H.; Li, Y.; Tian, A.; Tang, T.; Xu, Z.; Chen, X.; Hu, N.; Dong, W.; Li, Q.; and Chen, L

    Openai o1 system card.arXiv preprint arXiv:2412.16720. Li, H.; Li, Y.; Tian, A.; Tang, T.; Xu, Z.; Chen, X.; Hu, N.; Dong, W.; Li, Q.; and Chen, L. 2024a. A survey on large languagemodelaccelerationbasedonkvcachemanagement. arXiv preprint arXiv:2412.19442. Li, Y.; Huang, Y.; Y...

    2026 arXiv

  2. [14]

    Ning, Y.; Li, W.; Fang, J.; Tan, N.; and Liu, H

    Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825. Ning, Y.; Li, W.; Fang, J.; Tan, N.; and Liu, H

    arXiv

  3. [15]

    Ong,I.;Almahairi,A.;Wu,V.;Chiang,W.-L.;Wu,T.;Gon- zalez, J

    Not all thoughts are generated equal: Efficient llm reason- ing via multi-turn reinforcement learning.arXiv preprint arXiv:2505.11827. Ong,I.;Almahairi,A.;Wu,V.;Chiang,W.-L.;Wu,T.;Gon- zalez, J. E.; Kadous, M. W.; and Stoica, I

    arXiv

  4. [16]

    Qiao, Z.; Deng, Y.; Zeng, J.; Wang, D.; Wei, L.; Wang, G.; Meng, F.; Zhou, J.; Ren, J.; and Zhang, Y

    Routellm: Learning to route llms with preference data.arXiv preprint arXiv:2406.18665. Qiao, Z.; Deng, Y.; Zeng, J.; Wang, D.; Wei, L.; Wang, G.; Meng, F.; Zhou, J.; Ren, J.; and Zhang, Y

    arXiv

  5. [17]

    InProceedingsofthe2025ConferenceonEmpirical Methods in Natural Language Processing, 8021–8040

    Concise: Confidence-guidedcompressioninstep-by-stepefficientrea- soning. InProceedingsofthe2025ConferenceonEmpirical Methods in Natural Language Processing, 8021–8040. Qu, X.; Li, Y.; Su, Z.-C.; Sun, W.; Yan, J.; Liu, D.; Cui, G.; Liu,D.;Liang,S.;He,J.;etal.2025.Asurveyofeffic...

    2025

  6. [19]

    Shi,L.;Zhang,H.;Yao,Y.;Li,Z.;andZhao,H.2024

    The benefits of a concise chainofthoughtonproblem-solvinginlargelanguagemod- els.arXiv preprint arXiv:2401.05618. Shi,L.;Zhang,H.;Yao,Y.;Li,Z.;andZhao,H.2024. Keep the cost down: A review on methods to optimize LLM’s KV-cache consumption.arXiv preprint arXiv:2407.18003. Song, ...

    2024 arXiv

  7. [20]

    InPro- ceedings of the 63rd Annual Meeting of the Association for ComputationalLinguistics(Volume1:LongPapers),15629– 15644

    Irt-router: Effective and inter- pretable multi-llm routing via item response theory. InPro- ceedings of the 63rd Annual Meeting of the Association for ComputationalLinguistics(Volume1:LongPapers),15629– 15644. Tang,Y.;Jia,B.;Huang,W.;Zhang,L.;Xie,J.;Li,W.;Li,W.; Hu,J.;Ji,X.C....

    2026 arXiv

  8. [21]

    D2o: Dynamic discriminativeoperationsforefficientgenerativeinferenceof large language models.arXiv preprint arXiv:2406.13035,

    arXiv

  9. [22]

    Wait, we don’t need to" wait"! removing thinkingtokensimprovesreasoningefficiency.arXivpreprint arXiv:2506.08343,

    arXiv

  10. [23]

    Xu, J.; Zhu, Q.; Wu, Y.; Wang, Z.; Zhang, D.; Tian, M.; Duan,Y.;Li,S.;Wei,J.;Han,S.;etal.2026

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks.arXiv preprint arXiv:2407.08454. Xu, J.; Zhu, Q.; Wu, Y.; Wang, Z.; Zhang, D.; Tian, M.; Duan,Y.;Li,S.;Wei,J.;Han,S.;etal.2026. NanoResearch: Co-Evolving Skills, Memory, and Policy for Per...

    2026 arXiv

  11. [24]

    Yeo, E.; Tong, Y.; Niu, M.; Neubig, G.; and Yue, X

    IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Infer- ence.arXiv preprint arXiv:2605.25475. Yeo, E.; Tong, Y.; Niu, M.; Neubig, G.; and Yue, X

    arXiv

  12. [25]

    Yue, L.; Du, Y.; Wang, Y.; Gao, W.; Yao, F.; Wang, L.; Liu, Y.;Xu,Z.;Liu,Q.;Di,S.;etal.2025

    Demystifyinglongchain-of-thoughtreasoninginllms.arXiv preprint arXiv:2502.03373. Yue, L.; Du, Y.; Wang, Y.; Gao, W.; Yao, F.; Wang, L.; Liu, Y.;Xu,Z.;Liu,Q.;Di,S.;etal.2025. Don’tOverthinkIt:A SurveyofEfficientR1-styleLargeReasoningModels.arXiv preprint arXiv:2508.02120. Zandi...

    2025 arXiv

  13. [26]

    Zeng, Z.; Huang, X.; Li, B.; Zhang, H.; and Deng, Z

    Turboquant: Online vector quantization with near-optimal distortion rate.arXiv preprint arXiv:2504.19874. Zeng, Z.; Huang, X.; Li, B.; Zhang, H.; and Deng, Z

    arXiv

  14. [27]

    Zhang,Y.;Du,Y.;Luo,G.;Zhong,Y.;Zhang,Z.;Liu,S.;and Ji, R

    Doneisbetterthanperfect:Unlockingefficientreason- ing by structured multi-turn decomposition.arXiv preprint arXiv:2505.19788. Zhang,Y.;Du,Y.;Luo,G.;Zhong,Y.;Zhang,Z.;Liu,S.;and Ji, R

    arXiv

  15. [28]

    InForty-firstinternationalconferenceonmachine learning

    Cam: Cache merging for memory-efficient llms inference. InForty-firstinternationalconferenceonmachine learning. Zhu,Q.;Li,D.;Li,L.;Qin,X.;Li,W.;Gu,H.;Xu,H.;Han, S.;andGuo,Y.2026. Outliermatters:Efficientlong-to-short reasoningviaoutlier-guidedmodelmerging. InProceedings oftheA...

    2026

  16. [2021]

    arXiv preprint arXiv:2110.14168

    Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; et al

    arXiv

  17. [2023]

    Renze, M.; and Guven, E

    Gpqa: A graduate-levelgoogle-proofq&abenchmark.arXivpreprint arXiv:2311.12022. Renze, M.; and Guven, E

    arXiv

  18. [2024]

    Cao,C.;Li,M.;Dai,J.;Yang,J.;Zhao,Z.;Zhang,S.;Shi,W.; Liu,C.;Han,S.;andGuo,Y.2025

    Pyramidkv: Dy- namickvcachecompressionbasedonpyramidalinformation funneling.arXiv preprint arXiv:2406.02069. Cao,C.;Li,M.;Dai,J.;Yang,J.;Zhao,Z.;Zhang,S.;Shi,W.; Liu,C.;Han,S.;andGuo,Y.2025. Towardsadvancedmath- ematical reasoning for LLMs via first-order logic theorem proving...

    2025 arXiv

  19. [2025]

    arXiv preprint arXiv:2503.04697

    L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697. Cai, Z.; Xiao, W.; Sun, H.; Zhang, Y.; Wan, K.; Li, Y.; Zhou, Y.; Chang, L.-W.; Gu, J.; Dong, Z.; et al

    arXiv

  20. [2026]

    Fu,Y.;Chen,J.;Zhuang,Y.;Fu,Z.;Stoica,I.;andZhang,H

    Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.Advances in Neural Information Processing Systems, 38: 113152–113188. Fu,Y.;Chen,J.;Zhuang,Y.;Fu,Z.;Stoica,I.;andZhang,H. 2025.Reasoningwithoutself-doubt:Moreefficientchain-of- thoug...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.